sysinv 2020-05-16 17:16:50.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-16-50 patch sysinv 2020-05-16 17:16:50.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:21:51.013 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-21-50 patch sysinv 2020-05-16 17:21:51.013 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:26:50.945 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-26-50 patch sysinv 2020-05-16 17:26:50.946 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:31:50.996 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-31-50 patch sysinv 2020-05-16 17:31:50.997 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:36:51.525 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-36-51 patch sysinv 2020-05-16 17:36:51.526 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:41:50.960 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-41-50 patch sysinv 2020-05-16 17:41:50.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:46:50.987 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-46-50 patch sysinv 2020-05-16 17:46:50.987 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:51:51.019 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-51-50 patch sysinv 2020-05-16 17:51:51.019 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 17:56:51.124 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-17-56-51 patch sysinv 2020-05-16 17:56:51.124 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:01:51.181 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-01-51 patch sysinv 2020-05-16 18:01:51.182 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:06:51.202 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-06-51 patch sysinv 2020-05-16 18:06:51.203 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:11:51.417 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-11-51 patch sysinv 2020-05-16 18:11:51.418 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:15:23.223 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 18:16:51.263 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-16-51 patch sysinv 2020-05-16 18:16:51.263 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:16:53.231 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 18:16:53.241 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:17:53.302 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 18:17:53.303 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139843 sec sysinv 2020-05-16 18:17:53.312 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:18:53.381 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 18:18:53.390 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:19:53.451 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 18:19:53.452 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148584 sec sysinv 2020-05-16 18:19:53.462 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:20:53.528 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 18:20:53.537 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:21:51.311 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-21-51 patch sysinv 2020-05-16 18:21:51.311 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 18:21:53.597 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 18:21:53.598 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14624 sec sysinv 2020-05-16 18:21:53.610 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 18:22:03.435 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589652962.0-15405740428458/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589652962.0-15405740428458/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 18:22:03.436 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 18:26:51.302 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-26-51 patch sysinv 2020-05-16 18:26:51.303 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:31:51.396 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-31-51 patch sysinv 2020-05-16 18:31:51.396 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:36:51.274 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-36-51 patch sysinv 2020-05-16 18:36:51.275 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:41:51.472 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-41-51 patch sysinv 2020-05-16 18:41:51.473 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:46:51.447 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-46-51 patch sysinv 2020-05-16 18:46:51.448 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:51:51.532 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-51-51 patch sysinv 2020-05-16 18:51:51.533 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 18:56:51.398 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-18-56-51 patch sysinv 2020-05-16 18:56:51.399 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:01:51.609 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-01-51 patch sysinv 2020-05-16 19:01:51.610 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:06:51.579 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-06-51 patch sysinv 2020-05-16 19:06:51.580 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:11:51.659 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-11-51 patch sysinv 2020-05-16 19:11:51.659 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:16:51.569 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-16-51 patch sysinv 2020-05-16 19:16:51.570 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:21:51.722 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-21-51 patch sysinv 2020-05-16 19:21:51.722 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:22:03.440 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 19:23:57.127 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 19:23:57.135 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 19:24:57.211 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 19:24:57.212 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152548 sec sysinv 2020-05-16 19:24:57.232 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 19:25:57.295 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 19:25:57.304 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 19:26:51.661 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-26-51 patch sysinv 2020-05-16 19:26:51.662 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 19:26:57.365 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 19:26:57.366 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15323 sec sysinv 2020-05-16 19:26:57.374 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 19:27:57.434 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 19:27:57.444 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 19:28:40.727 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589656959.58-228546309531160/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589656959.58-228546309531160/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 19:28:40.728 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 19:28:42.786 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 45.419289 sec sysinv 2020-05-16 19:31:51.666 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-31-51 patch sysinv 2020-05-16 19:31:51.666 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:36:52.012 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-36-51 patch sysinv 2020-05-16 19:36:52.013 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:41:51.770 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-41-51 patch sysinv 2020-05-16 19:41:51.771 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:46:51.817 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-46-51 patch sysinv 2020-05-16 19:46:51.817 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:51:51.790 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-51-51 patch sysinv 2020-05-16 19:51:51.791 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 19:56:51.850 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-19-56-51 patch sysinv 2020-05-16 19:56:51.850 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:01:51.847 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-01-51 patch sysinv 2020-05-16 20:01:51.847 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:06:51.931 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-06-51 patch sysinv 2020-05-16 20:06:51.931 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:11:51.929 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-11-51 patch sysinv 2020-05-16 20:11:51.930 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:16:52.117 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-16-52 patch sysinv 2020-05-16 20:16:52.117 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:21:51.911 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-21-51 patch sysinv 2020-05-16 20:21:51.912 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:26:51.938 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-26-51 patch sysinv 2020-05-16 20:26:51.938 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:28:40.781 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 20:29:46.227 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 20:29:46.237 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:30:46.299 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 20:30:46.301 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150368 sec sysinv 2020-05-16 20:30:46.310 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:31:46.372 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 20:31:46.383 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:31:52.092 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-31-52 patch sysinv 2020-05-16 20:31:52.092 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:32:46.446 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 20:32:46.447 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146222 sec sysinv 2020-05-16 20:32:46.457 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:33:46.525 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 20:33:46.543 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 20:34:46.616 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 20:34:46.617 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.169714 sec sysinv 2020-05-16 20:34:46.631 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 20:35:22.021 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589660960.86-225488985538873/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589660960.86-225488985538873/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 20:35:22.022 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 20:36:52.052 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-36-52 patch sysinv 2020-05-16 20:36:52.053 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:41:52.142 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-41-52 patch sysinv 2020-05-16 20:41:52.144 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:46:52.144 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-46-52 patch sysinv 2020-05-16 20:46:52.144 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:51:52.264 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-51-52 patch sysinv 2020-05-16 20:51:52.265 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 20:56:52.259 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-20-56-52 patch sysinv 2020-05-16 20:56:52.259 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:01:52.307 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-01-52 patch sysinv 2020-05-16 21:01:52.308 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:06:52.278 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-06-52 patch sysinv 2020-05-16 21:06:52.279 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:11:52.345 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-11-52 patch sysinv 2020-05-16 21:11:52.346 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:16:52.320 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-16-52 patch sysinv 2020-05-16 21:16:52.321 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:21:52.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-21-52 patch sysinv 2020-05-16 21:21:52.409 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:26:52.405 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-26-52 patch sysinv 2020-05-16 21:26:52.406 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:31:52.499 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-31-52 patch sysinv 2020-05-16 21:31:52.500 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:35:22.053 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 21:36:50.101 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 21:36:50.115 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:36:52.366 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-36-52 patch sysinv 2020-05-16 21:36:52.367 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:37:50.188 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 21:37:50.189 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.175811 sec sysinv 2020-05-16 21:37:50.204 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:38:50.274 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 21:38:50.290 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:39:50.354 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 21:39:50.355 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.164983 sec sysinv 2020-05-16 21:39:50.365 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:40:50.433 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 21:40:50.447 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 21:41:50.519 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 21:41:50.520 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.164694 sec sysinv 2020-05-16 21:41:50.537 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 21:41:52.433 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-41-52 patch sysinv 2020-05-16 21:41:52.433 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:42:07.082 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589664966.01-76194517218906/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589664966.01-76194517218906/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 21:42:07.084 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 21:46:52.601 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-46-52 patch sysinv 2020-05-16 21:46:52.601 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:51:52.551 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-51-52 patch sysinv 2020-05-16 21:51:52.551 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 21:56:52.511 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-21-56-52 patch sysinv 2020-05-16 21:56:52.512 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:01:52.646 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-01-52 patch sysinv 2020-05-16 22:01:52.647 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:06:52.622 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-06-52 patch sysinv 2020-05-16 22:06:52.623 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:11:52.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-11-52 patch sysinv 2020-05-16 22:11:52.707 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:16:52.678 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-16-52 patch sysinv 2020-05-16 22:16:52.679 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:21:52.818 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-21-52 patch sysinv 2020-05-16 22:21:52.819 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:26:52.735 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-26-52 patch sysinv 2020-05-16 22:26:52.736 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:31:52.778 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-31-52 patch sysinv 2020-05-16 22:31:52.778 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:36:53.131 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-36-53 patch sysinv 2020-05-16 22:36:53.132 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:41:52.772 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-41-52 patch sysinv 2020-05-16 22:41:52.773 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:42:07.097 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 22:43:54.054 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 22:43:54.063 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 22:44:54.131 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 22:44:54.133 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144434 sec sysinv 2020-05-16 22:44:54.143 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 22:45:54.204 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 22:45:54.215 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 22:46:52.923 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-46-52 patch sysinv 2020-05-16 22:46:52.923 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 22:46:54.275 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 22:46:54.276 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14337 sec sysinv 2020-05-16 22:46:54.284 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 22:47:54.344 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 22:47:54.359 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 22:48:48.656 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589668967.58-124937365548637/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589668967.58-124937365548637/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 22:48:48.658 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 22:48:50.848 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 56.571212 sec sysinv 2020-05-16 22:51:52.889 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-51-52 patch sysinv 2020-05-16 22:51:52.890 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 22:56:52.955 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-22-56-52 patch sysinv 2020-05-16 22:56:52.956 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:01:52.963 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-01-52 patch sysinv 2020-05-16 23:01:52.964 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:06:53.173 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-06-53 patch sysinv 2020-05-16 23:06:53.174 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:11:52.960 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-11-52 patch sysinv 2020-05-16 23:11:52.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:16:53.071 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-16-53 patch sysinv 2020-05-16 23:16:53.071 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:21:53.031 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-21-52 patch sysinv 2020-05-16 23:21:53.032 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:26:53.057 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-26-53 patch sysinv 2020-05-16 23:26:53.057 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:31:53.291 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-31-53 patch sysinv 2020-05-16 23:31:53.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:36:53.698 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-36-53 patch sysinv 2020-05-16 23:36:53.698 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:41:53.196 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-41-53 patch sysinv 2020-05-16 23:41:53.197 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:46:53.270 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-46-53 patch sysinv 2020-05-16 23:46:53.270 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:48:48.710 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-16 23:49:54.298 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 23:49:54.308 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:50:54.373 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 23:50:54.374 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143704 sec sysinv 2020-05-16 23:50:54.387 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:51:53.344 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-51-53 patch sysinv 2020-05-16 23:51:53.344 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-16 23:51:54.453 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 23:51:54.472 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:52:54.534 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 23:52:54.535 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.160614 sec sysinv 2020-05-16 23:52:54.544 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:53:54.604 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-16 23:53:54.614 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-16 23:54:54.676 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-16 23:54:54.677 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141625 sec sysinv 2020-05-16 23:54:54.685 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-16 23:55:32.381 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589672971.07-215981502784607/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589672971.07-215981502784607/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-16 23:55:32.382 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-16 23:56:53.268 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-16-23-56-53 patch sysinv 2020-05-16 23:56:53.268 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:01:53.286 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-01-53 patch sysinv 2020-05-17 00:01:53.286 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:06:53.377 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-06-53 patch sysinv 2020-05-17 00:06:53.377 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:11:53.434 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-11-53 patch sysinv 2020-05-17 00:11:53.435 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:16:53.436 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-16-53 patch sysinv 2020-05-17 00:16:53.436 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:21:53.477 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-21-53 patch sysinv 2020-05-17 00:21:53.477 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:26:53.529 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-26-53 patch sysinv 2020-05-17 00:26:53.529 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:31:53.554 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-31-53 patch sysinv 2020-05-17 00:31:53.554 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:36:53.974 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-36-53 patch sysinv 2020-05-17 00:36:53.974 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:41:53.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-41-53 patch sysinv 2020-05-17 00:41:53.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:46:53.603 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-46-53 patch sysinv 2020-05-17 00:46:53.603 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:51:53.683 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-51-53 patch sysinv 2020-05-17 00:51:53.683 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:55:32.414 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 00:56:53.591 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-00-56-53 patch sysinv 2020-05-17 00:56:53.591 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 00:56:58.175 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 00:56:58.191 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 00:57:58.269 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 00:57:58.270 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.167021 sec sysinv 2020-05-17 00:57:58.283 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 00:58:58.358 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 00:58:58.370 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 00:59:58.432 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 00:59:58.433 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.162068 sec sysinv 2020-05-17 00:59:58.441 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 01:00:58.503 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 01:00:58.512 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 01:01:53.791 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-01-53 patch sysinv 2020-05-17 01:01:53.792 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 01:01:58.578 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 01:01:58.580 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146221 sec sysinv 2020-05-17 01:01:58.590 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 01:02:14.905 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589676973.7-154500257056619/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589676973.7-154500257056619/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 01:02:14.907 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 01:06:53.719 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-06-53 patch sysinv 2020-05-17 01:06:53.720 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:11:53.890 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-11-53 patch sysinv 2020-05-17 01:11:53.890 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:16:53.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-16-53 patch sysinv 2020-05-17 01:16:53.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:21:53.967 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-21-53 patch sysinv 2020-05-17 01:21:53.967 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:26:53.883 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-26-53 patch sysinv 2020-05-17 01:26:53.884 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:31:54.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-31-54 patch sysinv 2020-05-17 01:31:54.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:36:53.898 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-36-53 patch sysinv 2020-05-17 01:36:53.898 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:41:54.106 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-41-54 patch sysinv 2020-05-17 01:41:54.106 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:46:53.934 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-46-53 patch sysinv 2020-05-17 01:46:53.934 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:51:53.942 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-51-53 patch sysinv 2020-05-17 01:51:53.943 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 01:56:54.166 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-01-56-54 patch sysinv 2020-05-17 01:56:54.167 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:01:54.010 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-01-53 patch sysinv 2020-05-17 02:01:54.011 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:02:14.918 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 02:04:02.090 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 02:04:02.116 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 02:05:02.176 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 02:05:02.177 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.153501 sec sysinv 2020-05-17 02:05:02.187 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 02:06:02.248 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 02:06:02.258 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 02:06:54.151 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-06-54 patch sysinv 2020-05-17 02:06:54.151 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 02:07:02.321 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 02:07:02.322 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14424 sec sysinv 2020-05-17 02:07:02.336 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 02:08:02.409 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 02:08:02.420 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 02:08:53.014 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589680971.56-161243334612612/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589680971.56-161243334612612/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 02:08:53.015 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 02:08:54.941 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 52.618462 sec sysinv 2020-05-17 02:11:54.130 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-11-54 patch sysinv 2020-05-17 02:11:54.131 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:16:54.209 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-16-54 patch sysinv 2020-05-17 02:16:54.210 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:21:54.240 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-21-54 patch sysinv 2020-05-17 02:21:54.241 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:26:54.302 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-26-54 patch sysinv 2020-05-17 02:26:54.302 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:31:54.254 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-31-54 patch sysinv 2020-05-17 02:31:54.254 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:36:54.255 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-36-54 patch sysinv 2020-05-17 02:36:54.255 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:41:54.324 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-41-54 patch sysinv 2020-05-17 02:41:54.324 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:46:54.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-46-54 patch sysinv 2020-05-17 02:46:54.409 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:51:54.430 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-51-54 patch sysinv 2020-05-17 02:51:54.430 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 02:56:54.432 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-02-56-54 patch sysinv 2020-05-17 02:56:54.432 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:01:54.533 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-01-54 patch sysinv 2020-05-17 03:01:54.533 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:06:54.495 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-06-54 patch sysinv 2020-05-17 03:06:54.496 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:08:53.068 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 03:09:58.387 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 03:09:58.396 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:10:58.457 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 03:10:58.458 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147334 sec sysinv 2020-05-17 03:10:58.467 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:11:54.506 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-11-54 patch sysinv 2020-05-17 03:11:54.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:11:58.528 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 03:11:58.536 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:12:58.613 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 03:12:58.614 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.155547 sec sysinv 2020-05-17 03:12:58.623 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:13:58.695 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 03:13:58.703 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 03:14:58.765 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 03:14:58.766 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150999 sec sysinv 2020-05-17 03:14:58.774 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 03:15:31.320 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589684970.17-104420751785566/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589684970.17-104420751785566/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 03:15:31.321 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 03:16:54.538 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-16-54 patch sysinv 2020-05-17 03:16:54.538 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:21:54.561 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-21-54 patch sysinv 2020-05-17 03:21:54.561 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:26:54.633 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-26-54 patch sysinv 2020-05-17 03:26:54.633 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:31:54.622 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-31-54 patch sysinv 2020-05-17 03:31:54.622 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:36:55.030 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-36-55 patch sysinv 2020-05-17 03:36:55.031 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:41:54.599 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-41-54 patch sysinv 2020-05-17 03:41:54.600 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:46:54.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-46-54 patch sysinv 2020-05-17 03:46:54.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:51:54.665 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-51-54 patch sysinv 2020-05-17 03:51:54.665 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 03:56:54.739 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-03-56-54 patch sysinv 2020-05-17 03:56:54.739 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:01:54.811 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-01-54 patch sysinv 2020-05-17 04:01:54.811 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:06:54.823 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-06-54 patch sysinv 2020-05-17 04:06:54.823 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:11:54.892 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-11-54 patch sysinv 2020-05-17 04:11:54.893 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:15:31.349 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 04:16:54.861 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-16-54 patch sysinv 2020-05-17 04:16:54.862 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:17:02.267 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 04:17:02.276 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:18:02.343 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 04:18:02.344 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.149962 sec sysinv 2020-05-17 04:18:02.356 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:19:02.419 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 04:19:02.431 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:20:02.494 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 04:20:02.495 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.149807 sec sysinv 2020-05-17 04:20:02.502 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:21:02.564 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 04:21:02.572 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:21:54.973 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-21-54 patch sysinv 2020-05-17 04:21:54.974 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 04:22:02.646 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 04:22:02.647 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152232 sec sysinv 2020-05-17 04:22:02.663 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 04:22:15.980 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589688974.68-192747620766770/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589688974.68-192747620766770/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 04:22:15.981 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 04:26:54.994 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-26-54 patch sysinv 2020-05-17 04:26:54.994 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:31:55.069 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-31-55 patch sysinv 2020-05-17 04:31:55.069 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:36:54.952 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-36-54 patch sysinv 2020-05-17 04:36:54.953 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:41:55.102 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-41-55 patch sysinv 2020-05-17 04:41:55.102 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:46:55.004 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-46-54 patch sysinv 2020-05-17 04:46:55.004 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:51:55.039 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-51-55 patch sysinv 2020-05-17 04:51:55.040 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 04:56:55.065 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-04-56-55 patch sysinv 2020-05-17 04:56:55.065 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:01:55.121 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-01-55 patch sysinv 2020-05-17 05:01:55.121 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:06:55.237 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-06-55 patch sysinv 2020-05-17 05:06:55.238 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:11:55.172 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-11-55 patch sysinv 2020-05-17 05:11:55.172 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:16:55.348 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-16-55 patch sysinv 2020-05-17 05:16:55.349 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:21:55.315 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-21-55 patch sysinv 2020-05-17 05:21:55.316 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:22:15.990 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 05:24:06.179 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 05:24:06.188 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 05:25:06.250 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 05:25:06.251 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142496 sec sysinv 2020-05-17 05:25:06.270 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 05:26:06.341 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 05:26:06.349 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 05:26:55.372 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-26-55 patch sysinv 2020-05-17 05:26:55.372 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 05:27:06.411 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 05:27:06.412 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.159823 sec sysinv 2020-05-17 05:27:06.422 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 05:28:06.484 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 05:28:06.492 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 05:28:56.667 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589692975.52-72579067961696/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589692975.52-72579067961696/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 05:28:56.668 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 05:28:58.868 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 52.456126 sec sysinv 2020-05-17 05:31:55.618 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-31-55 patch sysinv 2020-05-17 05:31:55.619 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:36:55.756 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-36-55 patch sysinv 2020-05-17 05:36:55.756 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:41:55.406 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-41-55 patch sysinv 2020-05-17 05:41:55.406 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:46:55.419 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-46-55 patch sysinv 2020-05-17 05:46:55.419 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:51:55.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-51-55 patch sysinv 2020-05-17 05:51:55.508 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 05:56:55.451 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-05-56-55 patch sysinv 2020-05-17 05:56:55.452 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:01:55.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-01-55 patch sysinv 2020-05-17 06:01:55.607 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:06:55.546 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-06-55 patch sysinv 2020-05-17 06:06:55.547 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:11:55.647 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-11-55 patch sysinv 2020-05-17 06:11:55.648 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:16:55.583 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-16-55 patch sysinv 2020-05-17 06:16:55.584 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:21:55.752 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-21-55 patch sysinv 2020-05-17 06:21:55.752 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:26:55.669 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-26-55 patch sysinv 2020-05-17 06:26:55.669 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:28:56.721 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 06:30:02.319 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 06:30:02.328 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:31:02.389 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 06:31:02.390 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145435 sec sysinv 2020-05-17 06:31:02.401 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:31:55.769 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-31-55 patch sysinv 2020-05-17 06:31:55.770 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:32:02.464 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 06:32:02.472 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:33:02.533 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 06:33:02.534 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144037 sec sysinv 2020-05-17 06:33:02.542 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:34:02.603 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 06:34:02.616 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 06:35:02.678 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 06:35:02.679 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144599 sec sysinv 2020-05-17 06:35:02.688 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 06:35:40.473 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589696974.4-208061940045227/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589696974.4-208061940045227/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.49.33:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.49.33:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 06:35:40.474 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 06:36:55.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-36-55 patch sysinv 2020-05-17 06:36:55.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:41:55.828 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-41-55 patch sysinv 2020-05-17 06:41:55.828 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:46:55.786 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-46-55 patch sysinv 2020-05-17 06:46:55.786 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:51:55.899 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-51-55 patch sysinv 2020-05-17 06:51:55.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 06:56:55.861 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-06-56-55 patch sysinv 2020-05-17 06:56:55.862 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:01:55.965 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-01-55 patch sysinv 2020-05-17 07:01:55.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:06:55.942 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-06-55 patch sysinv 2020-05-17 07:06:55.942 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:11:56.024 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-11-56 patch sysinv 2020-05-17 07:11:56.025 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:16:55.980 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-16-55 patch sysinv 2020-05-17 07:16:55.980 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:21:56.083 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-21-56 patch sysinv 2020-05-17 07:21:56.084 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:26:55.983 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-26-55 patch sysinv 2020-05-17 07:26:55.984 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:31:56.010 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-31-55 patch sysinv 2020-05-17 07:31:56.010 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:35:40.506 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 07:36:56.067 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-36-56 patch sysinv 2020-05-17 07:36:56.067 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:37:06.163 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 07:37:06.175 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:38:06.238 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 07:38:06.239 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147092 sec sysinv 2020-05-17 07:38:06.250 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:39:06.312 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 07:39:06.322 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:40:06.384 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 07:40:06.385 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144983 sec sysinv 2020-05-17 07:40:06.396 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:41:06.477 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 07:41:06.495 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:41:56.145 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-41-56 patch sysinv 2020-05-17 07:41:56.146 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 07:42:06.569 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 07:42:06.572 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.186669 sec sysinv 2020-05-17 07:42:06.590 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 07:42:20.822 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589700979.54-236945013450292/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589700979.54-236945013450292/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 07:42:20.823 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 07:46:56.312 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-46-56 patch sysinv 2020-05-17 07:46:56.312 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:51:56.217 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-51-56 patch sysinv 2020-05-17 07:51:56.218 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 07:56:56.155 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-07-56-56 patch sysinv 2020-05-17 07:56:56.156 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:01:56.276 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-01-56 patch sysinv 2020-05-17 08:01:56.276 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:06:56.298 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-06-56 patch sysinv 2020-05-17 08:06:56.298 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:11:56.332 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-11-56 patch sysinv 2020-05-17 08:11:56.333 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:16:56.272 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-16-56 patch sysinv 2020-05-17 08:16:56.273 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:21:56.392 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-21-56 patch sysinv 2020-05-17 08:21:56.392 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:26:56.395 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-26-56 patch sysinv 2020-05-17 08:26:56.396 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:31:56.527 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-31-56 patch sysinv 2020-05-17 08:31:56.527 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:36:56.811 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-36-56 patch sysinv 2020-05-17 08:36:56.811 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:41:56.570 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-41-56 patch sysinv 2020-05-17 08:41:56.570 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:42:20.832 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 08:44:10.117 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 08:44:10.133 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 08:45:10.201 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 08:45:10.202 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15146 sec sysinv 2020-05-17 08:45:10.213 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 08:46:10.275 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 08:46:10.283 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 08:46:56.588 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-46-56 patch sysinv 2020-05-17 08:46:56.589 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 08:47:10.344 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 08:47:10.345 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142767 sec sysinv 2020-05-17 08:47:10.354 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 08:48:10.415 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 08:48:10.423 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 08:49:00.357 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589704979.23-236056026444250/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589704979.23-236056026444250/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 08:49:00.359 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 08:49:03.334 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 52.988661 sec sysinv 2020-05-17 08:51:56.659 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-51-56 patch sysinv 2020-05-17 08:51:56.660 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 08:56:56.588 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-08-56-56 patch sysinv 2020-05-17 08:56:56.589 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:01:56.715 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-01-56 patch sysinv 2020-05-17 09:01:56.716 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:06:56.702 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-06-56 patch sysinv 2020-05-17 09:06:56.702 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:11:56.773 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-11-56 patch sysinv 2020-05-17 09:11:56.774 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:16:56.699 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-16-56 patch sysinv 2020-05-17 09:16:56.700 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:21:56.852 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-21-56 patch sysinv 2020-05-17 09:21:56.852 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:26:56.783 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-26-56 patch sysinv 2020-05-17 09:26:56.784 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:31:56.777 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-31-56 patch sysinv 2020-05-17 09:31:56.778 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:36:57.140 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-36-57 patch sysinv 2020-05-17 09:36:57.141 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:41:56.792 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-41-56 patch sysinv 2020-05-17 09:41:56.793 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:46:56.822 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-46-56 patch sysinv 2020-05-17 09:46:56.822 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:49:00.411 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 09:50:06.771 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 09:50:06.780 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:51:06.843 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 09:51:06.844 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143766 sec sysinv 2020-05-17 09:51:06.859 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:51:56.893 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-51-56 patch sysinv 2020-05-17 09:51:56.894 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 09:52:06.928 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 09:52:06.937 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:53:06.997 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 09:53:06.998 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.153552 sec sysinv 2020-05-17 09:53:07.007 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:54:07.068 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 09:54:07.076 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 09:55:07.136 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 09:55:07.137 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137971 sec sysinv 2020-05-17 09:55:07.145 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 09:55:41.506 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589708980.43-40251529556668/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589708980.43-40251529556668/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 09:55:41.507 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 09:56:56.989 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-09-56-56 patch sysinv 2020-05-17 09:56:56.989 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:01:56.955 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-01-56 patch sysinv 2020-05-17 10:01:56.956 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:06:57.058 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-06-57 patch sysinv 2020-05-17 10:06:57.058 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:11:56.990 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-11-56 patch sysinv 2020-05-17 10:11:56.990 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:16:57.138 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-16-57 patch sysinv 2020-05-17 10:16:57.138 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:21:57.075 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-21-57 patch sysinv 2020-05-17 10:21:57.076 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:26:57.223 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-26-57 patch sysinv 2020-05-17 10:26:57.224 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:31:57.138 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-31-57 patch sysinv 2020-05-17 10:31:57.139 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:36:57.123 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-36-57 patch sysinv 2020-05-17 10:36:57.123 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:41:57.283 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-41-57 patch sysinv 2020-05-17 10:41:57.284 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:46:57.280 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-46-57 patch sysinv 2020-05-17 10:46:57.281 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:51:57.305 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-51-57 patch sysinv 2020-05-17 10:51:57.306 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:55:41.536 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 10:56:57.263 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-10-56-57 patch sysinv 2020-05-17 10:56:57.264 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 10:57:10.628 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 10:57:10.636 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 10:58:10.697 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 10:58:10.698 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137168 sec sysinv 2020-05-17 10:58:10.707 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 10:59:10.763 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 10:59:10.772 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 11:00:10.834 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 11:00:10.835 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137391 sec sysinv 2020-05-17 11:00:10.846 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 11:01:10.908 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 11:01:10.922 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 11:01:57.337 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-01-57 patch sysinv 2020-05-17 11:01:57.337 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 11:02:10.985 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 11:02:10.986 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150665 sec sysinv 2020-05-17 11:02:11.013 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 11:02:23.991 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589712982.85-130944039836396/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589712982.85-130944039836396/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 11:02:23.992 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 11:06:57.362 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-06-57 patch sysinv 2020-05-17 11:06:57.362 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:11:57.487 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-11-57 patch sysinv 2020-05-17 11:11:57.488 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:16:57.406 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-16-57 patch sysinv 2020-05-17 11:16:57.407 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:21:57.521 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-21-57 patch sysinv 2020-05-17 11:21:57.522 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:26:57.493 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-26-57 patch sysinv 2020-05-17 11:26:57.493 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:31:57.588 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-31-57 patch sysinv 2020-05-17 11:31:57.588 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:36:57.528 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-36-57 patch sysinv 2020-05-17 11:36:57.529 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:41:57.694 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-41-57 patch sysinv 2020-05-17 11:41:57.694 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:46:57.594 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-46-57 patch sysinv 2020-05-17 11:46:57.594 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:51:57.652 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-51-57 patch sysinv 2020-05-17 11:51:57.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 11:56:57.667 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-11-56-57 patch sysinv 2020-05-17 11:56:57.668 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:01:57.729 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-01-57 patch sysinv 2020-05-17 12:01:57.730 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:02:24.000 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 12:04:14.505 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 12:04:14.518 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 12:05:14.587 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 12:05:14.588 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150646 sec sysinv 2020-05-17 12:05:14.608 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 12:06:14.675 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 12:06:14.687 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 12:06:57.804 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-06-57 patch sysinv 2020-05-17 12:06:57.805 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 12:07:14.749 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 12:07:14.750 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.161621 sec sysinv 2020-05-17 12:07:14.760 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 12:08:14.827 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 12:08:14.843 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 12:09:03.757 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589716982.61-251165979823473/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589716982.61-251165979823473/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 12:09:03.758 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 12:09:05.876 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 51.124953 sec sysinv 2020-05-17 12:11:57.799 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-11-57 patch sysinv 2020-05-17 12:11:57.799 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:16:57.829 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-16-57 patch sysinv 2020-05-17 12:16:57.830 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:21:57.862 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-21-57 patch sysinv 2020-05-17 12:21:57.862 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:26:57.921 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-26-57 patch sysinv 2020-05-17 12:26:57.921 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:31:57.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-31-57 patch sysinv 2020-05-17 12:31:57.962 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:36:57.873 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-36-57 patch sysinv 2020-05-17 12:36:57.873 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:41:57.959 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-41-57 patch sysinv 2020-05-17 12:41:57.959 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:46:58.046 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-46-58 patch sysinv 2020-05-17 12:46:58.046 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:51:57.941 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-51-57 patch sysinv 2020-05-17 12:51:57.941 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 12:56:58.062 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-12-56-58 patch sysinv 2020-05-17 12:56:58.063 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:01:57.992 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-01-57 patch sysinv 2020-05-17 13:01:57.993 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:06:58.020 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-06-58 patch sysinv 2020-05-17 13:06:58.021 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:09:03.811 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 13:10:09.335 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 13:10:09.344 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:11:09.408 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 13:11:09.410 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.16146 sec sysinv 2020-05-17 13:11:09.425 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:11:58.105 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-11-58 patch sysinv 2020-05-17 13:11:58.106 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:12:09.486 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 13:12:09.494 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:13:09.555 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 13:13:09.556 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145621 sec sysinv 2020-05-17 13:13:09.564 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:14:09.631 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 13:14:09.650 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 13:15:09.722 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 13:15:09.723 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.166588 sec sysinv 2020-05-17 13:15:09.735 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 13:15:43.266 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589720982.1-176966577445123/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589720982.1-176966577445123/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 13:15:43.267 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 13:16:58.236 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-16-58 patch sysinv 2020-05-17 13:16:58.237 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:21:58.212 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-21-58 patch sysinv 2020-05-17 13:21:58.212 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:26:58.321 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-26-58 patch sysinv 2020-05-17 13:26:58.322 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:31:58.233 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-31-58 patch sysinv 2020-05-17 13:31:58.234 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:36:58.689 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-36-58 patch sysinv 2020-05-17 13:36:58.690 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:41:58.442 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-41-58 patch sysinv 2020-05-17 13:41:58.443 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:46:58.434 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-46-58 patch sysinv 2020-05-17 13:46:58.434 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:51:58.478 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-51-58 patch sysinv 2020-05-17 13:51:58.479 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 13:56:58.409 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-13-56-58 patch sysinv 2020-05-17 13:56:58.409 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:01:58.566 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-01-58 patch sysinv 2020-05-17 14:01:58.569 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:06:58.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-06-58 patch sysinv 2020-05-17 14:06:58.508 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:11:58.501 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-11-58 patch sysinv 2020-05-17 14:11:58.501 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:15:43.294 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 14:16:58.509 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-16-58 patch sysinv 2020-05-17 14:16:58.509 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:17:13.182 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 14:17:13.190 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:18:13.252 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 14:18:13.253 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.136217 sec sysinv 2020-05-17 14:18:13.262 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:19:13.323 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 14:19:13.338 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:20:13.400 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 14:20:13.401 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147813 sec sysinv 2020-05-17 14:20:13.409 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:21:13.469 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 14:21:13.477 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:21:58.622 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-21-58 patch sysinv 2020-05-17 14:21:58.623 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 14:22:13.538 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 14:22:13.539 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138332 sec sysinv 2020-05-17 14:22:13.548 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 14:22:24.321 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589724983.19-103784627173082/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589724983.19-103784627173082/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 14:22:24.322 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 14:26:58.682 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-26-58 patch sysinv 2020-05-17 14:26:58.683 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:31:58.635 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-31-58 patch sysinv 2020-05-17 14:31:58.636 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:36:58.579 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-36-58 patch sysinv 2020-05-17 14:36:58.580 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:41:58.761 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-41-58 patch sysinv 2020-05-17 14:41:58.762 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:46:58.739 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-46-58 patch sysinv 2020-05-17 14:46:58.740 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:51:58.816 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-51-58 patch sysinv 2020-05-17 14:51:58.816 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 14:56:58.700 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-14-56-58 patch sysinv 2020-05-17 14:56:58.701 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:01:58.784 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-01-58 patch sysinv 2020-05-17 15:01:58.785 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:06:58.851 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-06-58 patch sysinv 2020-05-17 15:06:58.852 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:11:58.917 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-11-58 patch sysinv 2020-05-17 15:11:58.918 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:16:58.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-16-58 patch sysinv 2020-05-17 15:16:58.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:21:58.953 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-21-58 patch sysinv 2020-05-17 15:21:58.954 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:22:24.328 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 15:24:17.076 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 15:24:17.084 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 15:25:17.147 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 15:25:17.148 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137413 sec sysinv 2020-05-17 15:25:17.161 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 15:26:17.233 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 15:26:17.242 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 15:26:58.983 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-26-58 patch sysinv 2020-05-17 15:26:58.983 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 15:27:17.303 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 15:27:17.304 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15535 sec sysinv 2020-05-17 15:27:17.312 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 15:28:17.373 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 15:28:17.385 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 15:29:01.867 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589728980.37-43205439307460/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589728980.37-43205439307460/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 15:29:01.868 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 15:29:03.708 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 46.403253 sec sysinv 2020-05-17 15:31:59.009 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-31-58 patch sysinv 2020-05-17 15:31:59.009 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:36:59.312 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-36-59 patch sysinv 2020-05-17 15:36:59.312 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:41:58.965 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-41-58 patch sysinv 2020-05-17 15:41:58.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:46:59.101 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-46-59 patch sysinv 2020-05-17 15:46:59.101 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:51:59.075 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-51-59 patch sysinv 2020-05-17 15:51:59.075 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 15:56:59.140 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-15-56-59 patch sysinv 2020-05-17 15:56:59.141 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:01:59.165 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-01-59 patch sysinv 2020-05-17 16:01:59.166 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:06:59.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-06-59 patch sysinv 2020-05-17 16:06:59.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:11:59.234 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-11-59 patch sysinv 2020-05-17 16:11:59.235 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:16:59.337 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-16-59 patch sysinv 2020-05-17 16:16:59.338 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:21:59.294 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-21-59 patch sysinv 2020-05-17 16:21:59.294 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:26:59.389 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-26-59 patch sysinv 2020-05-17 16:26:59.389 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:29:01.922 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 16:30:07.161 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 16:30:07.182 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:31:07.252 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 16:31:07.253 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.160926 sec sysinv 2020-05-17 16:31:07.263 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:31:59.336 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-31-59 patch sysinv 2020-05-17 16:31:59.336 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:32:07.326 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 16:32:07.334 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:33:07.402 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 16:33:07.403 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.149177 sec sysinv 2020-05-17 16:33:07.412 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:34:07.482 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 16:34:07.491 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 16:35:07.553 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 16:35:07.554 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15082 sec sysinv 2020-05-17 16:35:07.568 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 16:35:39.350 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589732978.19-69434230297070/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589732978.19-69434230297070/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 16:35:39.352 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 16:36:59.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-36-59 patch sysinv 2020-05-17 16:36:59.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:41:59.361 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-41-59 patch sysinv 2020-05-17 16:41:59.361 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:46:59.520 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-46-59 patch sysinv 2020-05-17 16:46:59.521 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:51:59.456 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-51-59 patch sysinv 2020-05-17 16:51:59.457 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 16:56:59.597 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-16-56-59 patch sysinv 2020-05-17 16:56:59.598 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:01:59.497 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-01-59 patch sysinv 2020-05-17 17:01:59.498 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:06:59.558 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-06-59 patch sysinv 2020-05-17 17:06:59.559 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:11:59.597 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-11-59 patch sysinv 2020-05-17 17:11:59.597 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:16:59.599 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-16-59 patch sysinv 2020-05-17 17:16:59.600 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:21:59.669 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-21-59 patch sysinv 2020-05-17 17:21:59.670 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:26:59.692 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-26-59 patch sysinv 2020-05-17 17:26:59.693 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:31:59.732 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-31-59 patch sysinv 2020-05-17 17:31:59.734 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:35:39.378 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 17:36:59.756 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-36-59 patch sysinv 2020-05-17 17:36:59.757 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:37:11.044 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 17:37:11.060 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:38:11.119 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 17:38:11.120 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144883 sec sysinv 2020-05-17 17:38:11.130 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:39:11.192 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 17:39:11.202 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:40:11.275 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 17:40:11.277 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.156151 sec sysinv 2020-05-17 17:40:11.291 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:41:11.353 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 17:41:11.363 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:41:59.845 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-41-59 patch sysinv 2020-05-17 17:41:59.846 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 17:42:11.425 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 17:42:11.426 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148905 sec sysinv 2020-05-17 17:42:11.435 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 17:42:22.831 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589736981.74-84448382128452/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589736981.74-84448382128452/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 17:42:22.832 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 17:46:59.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-46-59 patch sysinv 2020-05-17 17:46:59.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:51:59.916 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-51-59 patch sysinv 2020-05-17 17:51:59.916 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 17:56:59.865 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-17-56-59 patch sysinv 2020-05-17 17:56:59.866 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:01:59.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-01-59 patch sysinv 2020-05-17 18:01:59.967 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:06:59.947 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-06-59 patch sysinv 2020-05-17 18:06:59.947 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:12:00.090 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-12-00 patch sysinv 2020-05-17 18:12:00.090 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:16:59.911 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-16-59 patch sysinv 2020-05-17 18:16:59.911 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:22:00.071 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-22-00 patch sysinv 2020-05-17 18:22:00.072 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:26:59.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-26-59 patch sysinv 2020-05-17 18:26:59.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:31:59.999 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-31-59 patch sysinv 2020-05-17 18:31:59.999 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:37:00.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-37-00 patch sysinv 2020-05-17 18:37:00.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:42:00.107 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-42-00 patch sysinv 2020-05-17 18:42:00.107 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:42:22.837 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 18:44:14.960 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 18:44:14.970 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 18:45:15.031 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 18:45:15.033 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139934 sec sysinv 2020-05-17 18:45:15.047 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 18:46:15.115 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 18:46:15.128 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 18:47:00.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-47-00 patch sysinv 2020-05-17 18:47:00.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 18:47:15.189 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 18:47:15.190 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.156891 sec sysinv 2020-05-17 18:47:15.201 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 18:48:15.261 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 18:48:15.269 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 18:49:06.039 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589740984.77-228344436875631/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589740984.77-228344436875631/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 18:49:06.041 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 18:49:08.021 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 52.830412 sec sysinv 2020-05-17 18:52:00.202 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-52-00 patch sysinv 2020-05-17 18:52:00.202 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 18:57:00.149 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-18-57-00 patch sysinv 2020-05-17 18:57:00.149 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:02:00.278 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-02-00 patch sysinv 2020-05-17 19:02:00.279 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:07:00.278 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-07-00 patch sysinv 2020-05-17 19:07:00.279 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:12:00.330 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-12-00 patch sysinv 2020-05-17 19:12:00.330 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:17:00.320 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-17-00 patch sysinv 2020-05-17 19:17:00.321 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:22:00.376 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-22-00 patch sysinv 2020-05-17 19:22:00.376 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:27:00.472 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-27-00 patch sysinv 2020-05-17 19:27:00.473 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:32:00.501 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-32-00 patch sysinv 2020-05-17 19:32:00.502 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:37:00.838 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-37-00 patch sysinv 2020-05-17 19:37:00.839 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:42:00.468 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-42-00 patch sysinv 2020-05-17 19:42:00.469 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:47:00.569 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-47-00 patch sysinv 2020-05-17 19:47:00.570 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:49:06.093 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 19:50:11.453 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 19:50:11.471 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:51:11.536 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 19:51:11.537 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152454 sec sysinv 2020-05-17 19:51:11.547 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:52:00.546 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-52-00 patch sysinv 2020-05-17 19:52:00.546 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 19:52:11.608 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 19:52:11.617 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:53:11.681 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 19:53:11.683 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145482 sec sysinv 2020-05-17 19:53:11.698 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:54:11.758 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 19:54:11.768 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 19:55:11.829 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 19:55:11.830 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147129 sec sysinv 2020-05-17 19:55:11.849 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 19:55:43.472 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589744982.42-122601800403483/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589744982.42-122601800403483/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 19:55:43.473 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 19:57:00.602 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-19-57-00 patch sysinv 2020-05-17 19:57:00.602 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:02:00.596 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-02-00 patch sysinv 2020-05-17 20:02:00.597 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:07:00.681 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-07-00 patch sysinv 2020-05-17 20:07:00.681 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:12:00.662 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-12-00 patch sysinv 2020-05-17 20:12:00.663 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:17:00.764 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-17-00 patch sysinv 2020-05-17 20:17:00.765 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:22:00.852 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-22-00 patch sysinv 2020-05-17 20:22:00.852 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:27:00.832 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-27-00 patch sysinv 2020-05-17 20:27:00.833 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:32:00.835 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-32-00 patch sysinv 2020-05-17 20:32:00.836 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:37:01.225 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-37-01 patch sysinv 2020-05-17 20:37:01.225 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:42:00.787 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-42-00 patch sysinv 2020-05-17 20:42:00.788 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:47:00.829 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-47-00 patch sysinv 2020-05-17 20:47:00.830 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:52:00.865 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-52-00 patch sysinv 2020-05-17 20:52:00.866 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:55:43.498 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 20:57:00.957 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-20-57-00 patch sysinv 2020-05-17 20:57:00.958 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 20:57:15.321 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 20:57:15.330 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 20:58:15.392 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 20:58:15.393 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139658 sec sysinv 2020-05-17 20:58:15.402 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 20:59:15.460 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 20:59:15.473 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 21:00:15.535 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 21:00:15.536 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142971 sec sysinv 2020-05-17 21:00:15.545 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 21:01:15.608 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 21:01:15.617 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 21:02:01.004 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-02-00 patch sysinv 2020-05-17 21:02:01.005 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 21:02:15.680 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 21:02:15.681 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144419 sec sysinv 2020-05-17 21:02:15.690 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 21:02:23.881 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589748982.73-166685232342684/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589748982.73-166685232342684/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 21:02:23.883 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 21:07:01.144 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-07-01 patch sysinv 2020-05-17 21:07:01.145 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:12:00.993 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-12-00 patch sysinv 2020-05-17 21:12:00.993 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:17:01.143 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-17-01 patch sysinv 2020-05-17 21:17:01.144 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:22:01.129 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-22-01 patch sysinv 2020-05-17 21:22:01.130 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:27:01.258 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-27-01 patch sysinv 2020-05-17 21:27:01.258 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:32:01.173 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-32-01 patch sysinv 2020-05-17 21:32:01.173 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:37:01.133 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-37-01 patch sysinv 2020-05-17 21:37:01.134 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:42:01.380 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-42-01 patch sysinv 2020-05-17 21:42:01.380 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:47:01.281 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-47-01 patch sysinv 2020-05-17 21:47:01.281 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:52:01.362 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-52-01 patch sysinv 2020-05-17 21:52:01.362 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 21:57:01.343 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-21-57-01 patch sysinv 2020-05-17 21:57:01.343 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:02:01.465 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-02-01 patch sysinv 2020-05-17 22:02:01.466 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:02:23.886 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 22:04:19.203 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 22:04:19.214 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 22:05:19.276 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 22:05:19.277 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138818 sec sysinv 2020-05-17 22:05:19.292 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 22:06:19.355 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 22:06:19.365 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 22:07:01.412 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-07-01 patch sysinv 2020-05-17 22:07:01.413 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 22:07:19.427 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 22:07:19.428 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150527 sec sysinv 2020-05-17 22:07:19.438 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 22:08:19.502 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 22:08:19.514 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 22:09:01.893 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589752980.63-175777938543441/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589752980.63-175777938543441/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 22:09:01.895 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 22:09:03.738 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 44.309611 sec sysinv 2020-05-17 22:12:01.549 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-12-01 patch sysinv 2020-05-17 22:12:01.549 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:17:01.497 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-17-01 patch sysinv 2020-05-17 22:17:01.498 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:22:01.496 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-22-01 patch sysinv 2020-05-17 22:22:01.497 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:27:01.674 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-27-01 patch sysinv 2020-05-17 22:27:01.674 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:32:01.534 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-32-01 patch sysinv 2020-05-17 22:32:01.534 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:37:01.590 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-37-01 patch sysinv 2020-05-17 22:37:01.591 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:42:01.642 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-42-01 patch sysinv 2020-05-17 22:42:01.643 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:47:01.729 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-47-01 patch sysinv 2020-05-17 22:47:01.729 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:52:01.766 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-52-01 patch sysinv 2020-05-17 22:52:01.767 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 22:57:01.702 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-22-57-01 patch sysinv 2020-05-17 22:57:01.704 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:02:01.809 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-02-01 patch sysinv 2020-05-17 23:02:01.810 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:07:01.893 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-07-01 patch sysinv 2020-05-17 23:07:01.894 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:09:01.948 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-17 23:10:07.190 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 23:10:07.199 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:11:07.260 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 23:11:07.262 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140478 sec sysinv 2020-05-17 23:11:07.276 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:12:01.898 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-12-01 patch sysinv 2020-05-17 23:12:01.899 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:12:07.339 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 23:12:07.352 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:13:07.427 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 23:13:07.428 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.16623 sec sysinv 2020-05-17 23:13:07.437 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:14:07.497 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-17 23:14:07.506 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-17 23:15:07.567 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-17 23:15:07.568 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139244 sec sysinv 2020-05-17 23:15:07.578 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-17 23:15:46.307 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589756984.9-266352940810444/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589756984.9-266352940810444/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-17 23:15:46.308 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-17 23:17:01.803 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-17-01 patch sysinv 2020-05-17 23:17:01.803 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:22:01.983 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-22-01 patch sysinv 2020-05-17 23:22:01.983 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:27:01.855 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-27-01 patch sysinv 2020-05-17 23:27:01.855 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:32:01.994 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-32-01 patch sysinv 2020-05-17 23:32:01.994 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:37:02.276 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-37-02 patch sysinv 2020-05-17 23:37:02.276 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:42:01.895 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-42-01 patch sysinv 2020-05-17 23:42:01.895 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:47:01.924 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-47-01 patch sysinv 2020-05-17 23:47:01.924 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:52:01.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-52-01 patch sysinv 2020-05-17 23:52:01.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-17 23:57:02.026 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-17-23-57-02 patch sysinv 2020-05-17 23:57:02.027 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:02:02.011 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-02-01 patch sysinv 2020-05-18 00:02:02.012 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:07:02.047 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-07-02 patch sysinv 2020-05-18 00:07:02.048 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:12:02.082 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-12-02 patch sysinv 2020-05-18 00:12:02.082 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:15:46.342 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 00:17:02.176 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-17-02 patch sysinv 2020-05-18 00:17:02.176 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:17:11.056 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 00:17:11.064 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:18:11.127 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 00:18:11.128 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.13874 sec sysinv 2020-05-18 00:18:11.136 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:19:11.197 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 00:19:11.205 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:20:11.268 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 00:20:11.269 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141104 sec sysinv 2020-05-18 00:20:11.284 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:21:11.355 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 00:21:11.369 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:22:02.433 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-22-02 patch sysinv 2020-05-18 00:22:02.434 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 00:22:11.430 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 00:22:11.431 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.161656 sec sysinv 2020-05-18 00:22:11.446 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 00:22:23.616 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589760982.5-163062079929815/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589760982.5-163062079929815/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 00:22:23.617 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 00:27:02.300 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-27-02 patch sysinv 2020-05-18 00:27:02.301 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:32:02.291 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-32-02 patch sysinv 2020-05-18 00:32:02.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:37:02.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-37-02 patch sysinv 2020-05-18 00:37:02.706 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:42:02.322 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-42-02 patch sysinv 2020-05-18 00:42:02.323 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:47:02.409 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-47-02 patch sysinv 2020-05-18 00:47:02.410 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:52:02.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-52-02 patch sysinv 2020-05-18 00:52:02.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 00:57:02.386 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-00-57-02 patch sysinv 2020-05-18 00:57:02.386 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:02:02.486 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-02-02 patch sysinv 2020-05-18 01:02:02.487 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:07:02.494 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-07-02 patch sysinv 2020-05-18 01:07:02.494 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:12:02.634 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-12-02 patch sysinv 2020-05-18 01:12:02.634 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:17:02.489 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-17-02 patch sysinv 2020-05-18 01:17:02.490 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:22:02.714 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-22-02 patch sysinv 2020-05-18 01:22:02.714 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:22:23.624 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 01:24:14.957 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 01:24:14.966 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 01:25:15.029 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 01:25:15.031 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14279 sec sysinv 2020-05-18 01:25:15.044 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 01:26:15.114 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 01:26:15.128 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 01:27:02.611 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-27-02 patch sysinv 2020-05-18 01:27:02.611 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 01:27:15.192 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 01:27:15.194 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.162618 sec sysinv 2020-05-18 01:27:15.205 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 01:28:15.265 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 01:28:15.273 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 01:29:03.824 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589764982.77-113922187523667/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589764982.77-113922187523667/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 01:29:03.827 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 01:29:05.808 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 50.614083 sec sysinv 2020-05-18 01:32:02.721 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-32-02 patch sysinv 2020-05-18 01:32:02.721 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:37:03.018 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-37-02 patch sysinv 2020-05-18 01:37:03.019 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:42:02.801 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-42-02 patch sysinv 2020-05-18 01:42:02.801 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:47:02.730 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-47-02 patch sysinv 2020-05-18 01:47:02.731 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:52:02.928 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-52-02 patch sysinv 2020-05-18 01:52:02.929 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 01:57:02.779 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-01-57-02 patch sysinv 2020-05-18 01:57:02.780 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:02:02.840 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-02-02 patch sysinv 2020-05-18 02:02:02.841 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:07:02.923 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-07-02 patch sysinv 2020-05-18 02:07:02.923 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:12:02.909 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-12-02 patch sysinv 2020-05-18 02:12:02.909 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:17:02.961 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-17-02 patch sysinv 2020-05-18 02:17:02.962 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:22:02.995 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-22-02 patch sysinv 2020-05-18 02:22:02.996 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:27:03.031 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-27-03 patch sysinv 2020-05-18 02:27:03.032 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:29:03.880 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 02:30:09.249 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 02:30:09.256 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:31:09.318 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 02:31:09.319 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137168 sec sysinv 2020-05-18 02:31:09.327 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:32:02.938 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-32-02 patch sysinv 2020-05-18 02:32:02.939 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:32:09.389 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 02:32:09.400 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:33:09.460 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 02:33:09.461 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142192 sec sysinv 2020-05-18 02:33:09.469 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:34:09.529 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 02:34:09.537 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 02:35:09.597 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 02:35:09.598 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.136935 sec sysinv 2020-05-18 02:35:09.606 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 02:35:46.070 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589768984.94-187175192720902/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589768984.94-187175192720902/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 02:35:46.072 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 02:37:03.086 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-37-03 patch sysinv 2020-05-18 02:37:03.087 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:42:03.004 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-42-02 patch sysinv 2020-05-18 02:42:03.005 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:47:03.044 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-47-03 patch sysinv 2020-05-18 02:47:03.044 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:52:03.182 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-52-03 patch sysinv 2020-05-18 02:52:03.183 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 02:57:03.278 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-02-57-03 patch sysinv 2020-05-18 02:57:03.278 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:02:03.226 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-02-03 patch sysinv 2020-05-18 03:02:03.227 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:07:03.232 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-07-03 patch sysinv 2020-05-18 03:07:03.232 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:12:03.269 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-12-03 patch sysinv 2020-05-18 03:12:03.269 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:17:03.256 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-17-03 patch sysinv 2020-05-18 03:17:03.256 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:22:03.385 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-22-03 patch sysinv 2020-05-18 03:22:03.386 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:27:03.338 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-27-03 patch sysinv 2020-05-18 03:27:03.338 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:32:03.405 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-32-03 patch sysinv 2020-05-18 03:32:03.406 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:35:46.104 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 03:37:03.398 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-37-03 patch sysinv 2020-05-18 03:37:03.399 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:37:13.094 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 03:37:13.102 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:38:13.161 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 03:38:13.162 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.13543 sec sysinv 2020-05-18 03:38:13.171 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:39:13.233 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 03:39:13.240 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:40:13.300 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 03:40:13.301 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138763 sec sysinv 2020-05-18 03:40:13.310 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:41:13.370 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 03:41:13.378 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:42:03.455 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-42-03 patch sysinv 2020-05-18 03:42:03.455 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 03:42:13.438 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 03:42:13.439 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137977 sec sysinv 2020-05-18 03:42:13.449 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 03:42:24.010 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589772982.92-10370965694829/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589772982.92-10370965694829/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 03:42:24.011 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 03:47:03.475 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-47-03 patch sysinv 2020-05-18 03:47:03.475 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:52:03.592 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-52-03 patch sysinv 2020-05-18 03:52:03.593 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 03:57:03.452 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-03-57-03 patch sysinv 2020-05-18 03:57:03.453 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:02:03.623 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-02-03 patch sysinv 2020-05-18 04:02:03.623 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:07:03.611 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-07-03 patch sysinv 2020-05-18 04:07:03.612 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:12:03.716 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-12-03 patch sysinv 2020-05-18 04:12:03.716 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:17:03.572 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-17-03 patch sysinv 2020-05-18 04:17:03.572 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:22:03.786 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-22-03 patch sysinv 2020-05-18 04:22:03.786 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:27:03.710 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-27-03 patch sysinv 2020-05-18 04:27:03.711 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:32:03.834 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-32-03 patch sysinv 2020-05-18 04:32:03.835 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:37:04.135 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-37-04 patch sysinv 2020-05-18 04:37:04.135 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:42:03.727 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-42-03 patch sysinv 2020-05-18 04:42:03.728 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:42:24.016 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 04:44:16.963 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 04:44:16.971 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 04:45:17.032 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 04:45:17.034 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.135605 sec sysinv 2020-05-18 04:45:17.053 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 04:46:17.116 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 04:46:17.124 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 04:47:04.036 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-47-03 patch sysinv 2020-05-18 04:47:04.037 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 04:47:17.184 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 04:47:17.185 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15069 sec sysinv 2020-05-18 04:47:17.193 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 04:48:17.255 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 04:48:17.266 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 04:49:02.148 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589776980.86-167517046422061/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589776980.86-167517046422061/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 04:49:02.149 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 04:49:04.708 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 47.522916 sec sysinv 2020-05-18 04:52:03.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-52-03 patch sysinv 2020-05-18 04:52:03.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 04:57:03.910 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-04-57-03 patch sysinv 2020-05-18 04:57:03.911 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:02:03.966 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-02-03 patch sysinv 2020-05-18 05:02:03.967 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:07:03.984 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-07-03 patch sysinv 2020-05-18 05:07:03.984 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:12:04.067 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-12-04 patch sysinv 2020-05-18 05:12:04.067 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:17:04.064 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-17-04 patch sysinv 2020-05-18 05:17:04.065 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:22:03.988 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-22-03 patch sysinv 2020-05-18 05:22:03.988 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:27:04.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-27-03 patch sysinv 2020-05-18 05:27:04.028 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:32:04.148 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-32-04 patch sysinv 2020-05-18 05:32:04.149 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:37:04.623 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-37-04 patch sysinv 2020-05-18 05:37:04.624 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:42:04.202 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-42-04 patch sysinv 2020-05-18 05:42:04.202 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:47:04.224 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-47-04 patch sysinv 2020-05-18 05:47:04.224 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:49:02.201 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 05:50:08.152 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 05:50:08.179 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:51:08.244 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 05:51:08.246 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.166304 sec sysinv 2020-05-18 05:51:08.257 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:52:04.291 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-52-04 patch sysinv 2020-05-18 05:52:04.291 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 05:52:08.319 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 05:52:08.327 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:53:08.388 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 05:53:08.389 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14343 sec sysinv 2020-05-18 05:53:08.405 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:54:08.467 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 05:54:08.475 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 05:55:08.537 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 05:55:08.539 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148905 sec sysinv 2020-05-18 05:55:08.550 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 05:55:42.337 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589780981.17-160361258756971/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589780981.17-160361258756971/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 05:55:42.338 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 05:57:04.286 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-05-57-04 patch sysinv 2020-05-18 05:57:04.286 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:02:04.361 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-02-04 patch sysinv 2020-05-18 06:02:04.362 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:07:04.320 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-07-04 patch sysinv 2020-05-18 06:07:04.321 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:12:04.381 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-12-04 patch sysinv 2020-05-18 06:12:04.381 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:17:04.364 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-17-04 patch sysinv 2020-05-18 06:17:04.364 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:22:04.506 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-22-04 patch sysinv 2020-05-18 06:22:04.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:27:04.442 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-27-04 patch sysinv 2020-05-18 06:27:04.442 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:32:04.553 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-32-04 patch sysinv 2020-05-18 06:32:04.553 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:37:04.787 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-37-04 patch sysinv 2020-05-18 06:37:04.788 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:42:04.605 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-42-04 patch sysinv 2020-05-18 06:42:04.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:47:04.553 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-47-04 patch sysinv 2020-05-18 06:47:04.554 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:52:04.652 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-52-04 patch sysinv 2020-05-18 06:52:04.652 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:55:42.367 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 06:57:04.556 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-06-57-04 patch sysinv 2020-05-18 06:57:04.556 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 06:57:12.032 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 06:57:12.040 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 06:58:12.108 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 06:58:12.109 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144282 sec sysinv 2020-05-18 06:58:12.118 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 06:59:12.184 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 06:59:12.193 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 07:00:12.255 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 07:00:12.256 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147332 sec sysinv 2020-05-18 07:00:12.266 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 07:01:12.324 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 07:01:12.332 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 07:02:04.752 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-02-04 patch sysinv 2020-05-18 07:02:04.753 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 07:02:12.392 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 07:02:12.393 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.136645 sec sysinv 2020-05-18 07:02:12.403 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 07:02:20.943 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589784979.7-122386098409855/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589784979.7-122386098409855/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 07:02:20.945 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 07:07:04.679 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-07-04 patch sysinv 2020-05-18 07:07:04.679 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:12:04.811 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-12-04 patch sysinv 2020-05-18 07:12:04.812 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:17:04.679 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-17-04 patch sysinv 2020-05-18 07:17:04.680 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:22:04.846 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-22-04 patch sysinv 2020-05-18 07:22:04.846 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:27:04.815 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-27-04 patch sysinv 2020-05-18 07:27:04.816 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:32:04.900 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-32-04 patch sysinv 2020-05-18 07:32:04.901 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:37:04.796 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-37-04 patch sysinv 2020-05-18 07:37:04.797 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:42:04.970 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-42-04 patch sysinv 2020-05-18 07:42:04.971 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:47:04.962 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-47-04 patch sysinv 2020-05-18 07:47:04.963 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:52:05.100 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-52-05 patch sysinv 2020-05-18 07:52:05.100 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 07:57:05.018 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-07-57-04 patch sysinv 2020-05-18 07:57:05.019 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:02:05.153 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-02-05 patch sysinv 2020-05-18 08:02:05.154 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:02:20.947 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 08:04:15.928 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 08:04:15.938 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 08:05:16.000 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 08:05:16.001 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140768 sec sysinv 2020-05-18 08:05:16.011 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 08:06:16.081 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 08:06:16.090 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 08:07:04.995 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-07-04 patch sysinv 2020-05-18 08:07:04.996 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 08:07:16.174 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 08:07:16.175 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.173477 sec sysinv 2020-05-18 08:07:16.184 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 08:08:16.246 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 08:08:16.255 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 08:08:58.751 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589788977.59-27187739834169/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589788977.59-27187739834169/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 08:08:58.752 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 08:09:00.596 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 44.420935 sec sysinv 2020-05-18 08:12:05.034 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-12-04 patch sysinv 2020-05-18 08:12:05.035 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:17:05.131 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-17-05 patch sysinv 2020-05-18 08:17:05.132 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:22:05.165 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-22-05 patch sysinv 2020-05-18 08:22:05.166 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:27:05.218 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-27-05 patch sysinv 2020-05-18 08:27:05.219 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:32:05.185 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-32-05 patch sysinv 2020-05-18 08:32:05.186 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:37:05.197 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-37-05 patch sysinv 2020-05-18 08:37:05.198 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:42:05.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-42-05 patch sysinv 2020-05-18 08:42:05.292 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:47:05.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-47-05 patch sysinv 2020-05-18 08:47:05.408 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:52:05.365 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-52-05 patch sysinv 2020-05-18 08:52:05.365 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 08:57:05.359 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-08-57-05 patch sysinv 2020-05-18 08:57:05.360 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:02:05.410 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-02-05 patch sysinv 2020-05-18 09:02:05.411 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:07:05.462 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-07-05 patch sysinv 2020-05-18 09:07:05.462 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:08:58.804 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 09:10:04.040 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 09:10:04.049 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:11:04.117 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 09:11:04.119 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146394 sec sysinv 2020-05-18 09:11:04.146 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:12:04.207 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 09:12:04.216 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:12:05.485 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-12-05 patch sysinv 2020-05-18 09:12:05.485 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:13:04.277 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 09:13:04.278 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.158917 sec sysinv 2020-05-18 09:13:04.286 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:14:04.346 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 09:14:04.360 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 09:15:04.421 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 09:15:04.422 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144009 sec sysinv 2020-05-18 09:15:04.431 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 09:15:38.258 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589792977.07-70911783639944/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589792977.07-70911783639944/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 09:15:38.260 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 09:17:05.533 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-17-05 patch sysinv 2020-05-18 09:17:05.534 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:22:05.522 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-22-05 patch sysinv 2020-05-18 09:22:05.523 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:27:05.542 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-27-05 patch sysinv 2020-05-18 09:27:05.543 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:32:05.737 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-32-05 patch sysinv 2020-05-18 09:32:05.738 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:37:05.988 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-37-05 patch sysinv 2020-05-18 09:37:05.989 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:42:05.631 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-42-05 patch sysinv 2020-05-18 09:42:05.632 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:47:05.703 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-47-05 patch sysinv 2020-05-18 09:47:05.704 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:52:05.689 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-52-05 patch sysinv 2020-05-18 09:52:05.689 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 09:57:05.741 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-09-57-05 patch sysinv 2020-05-18 09:57:05.742 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:02:05.733 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-02-05 patch sysinv 2020-05-18 10:02:05.734 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:07:05.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-07-05 patch sysinv 2020-05-18 10:07:05.825 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:12:05.780 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-12-05 patch sysinv 2020-05-18 10:12:05.780 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:15:38.288 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 10:17:05.879 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-17-05 patch sysinv 2020-05-18 10:17:05.880 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:17:07.924 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 10:17:07.934 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:18:07.996 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 10:18:07.997 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142463 sec sysinv 2020-05-18 10:18:08.010 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:19:08.072 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 10:19:08.086 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:20:08.165 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 10:20:08.166 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.168567 sec sysinv 2020-05-18 10:20:08.178 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:21:08.239 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 10:21:08.253 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:22:05.883 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-22-05 patch sysinv 2020-05-18 10:22:05.884 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 10:22:08.316 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 10:22:08.317 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150925 sec sysinv 2020-05-18 10:22:08.327 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 10:22:20.450 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589796979.24-97210553030975/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589796979.24-97210553030975/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 10:22:20.451 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 10:27:05.988 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-27-05 patch sysinv 2020-05-18 10:27:05.989 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:32:05.956 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-32-05 patch sysinv 2020-05-18 10:32:05.957 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:37:05.902 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-37-05 patch sysinv 2020-05-18 10:37:05.902 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:42:06.014 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-42-05 patch sysinv 2020-05-18 10:42:06.014 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:47:06.042 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-47-06 patch sysinv 2020-05-18 10:47:06.042 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:52:05.987 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-52-05 patch sysinv 2020-05-18 10:52:05.988 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 10:57:06.019 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-10-57-05 patch sysinv 2020-05-18 10:57:06.020 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:02:06.050 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-02-06 patch sysinv 2020-05-18 11:02:06.050 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:07:06.206 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-07-06 patch sysinv 2020-05-18 11:07:06.207 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:12:06.275 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-12-06 patch sysinv 2020-05-18 11:12:06.276 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:17:06.245 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-17-06 patch sysinv 2020-05-18 11:17:06.245 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:22:06.228 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-22-06 patch sysinv 2020-05-18 11:22:06.228 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:22:20.457 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 11:24:11.840 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 11:24:11.850 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 11:25:11.930 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 11:25:11.931 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.158042 sec sysinv 2020-05-18 11:25:11.940 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 11:26:12.008 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 11:26:12.016 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 11:27:06.334 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-27-06 patch sysinv 2020-05-18 11:27:06.334 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 11:27:12.076 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 11:27:12.078 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146553 sec sysinv 2020-05-18 11:27:12.086 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 11:28:12.151 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 11:28:12.164 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 11:28:58.543 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589800977.35-73801653056235/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589800977.35-73801653056235/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 11:28:58.544 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 11:29:00.369 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 48.290876 sec sysinv 2020-05-18 11:32:06.298 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-32-06 patch sysinv 2020-05-18 11:32:06.298 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:37:06.726 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-37-06 patch sysinv 2020-05-18 11:37:06.726 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:42:06.344 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-42-06 patch sysinv 2020-05-18 11:42:06.344 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:47:06.384 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-47-06 patch sysinv 2020-05-18 11:47:06.384 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:52:06.482 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-52-06 patch sysinv 2020-05-18 11:52:06.483 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 11:57:06.422 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-11-57-06 patch sysinv 2020-05-18 11:57:06.422 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:02:06.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-02-06 patch sysinv 2020-05-18 12:02:06.507 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:07:06.587 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-07-06 patch sysinv 2020-05-18 12:07:06.591 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:12:06.600 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-12-06 patch sysinv 2020-05-18 12:12:06.602 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:17:06.606 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-17-06 patch sysinv 2020-05-18 12:17:06.607 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:22:06.685 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-22-06 patch sysinv 2020-05-18 12:22:06.685 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:27:06.742 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-27-06 patch sysinv 2020-05-18 12:27:06.748 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:28:58.597 90717 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 12:30:03.827 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 12:30:03.838 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:31:03.909 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 12:31:03.912 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.151393 sec sysinv 2020-05-18 12:31:03.926 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:32:03.995 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 12:32:04.010 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:32:06.767 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-32-06 patch sysinv 2020-05-18 12:32:06.768 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:33:04.069 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 12:33:04.070 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.158125 sec sysinv 2020-05-18 12:33:04.078 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:34:04.138 10084 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 12:34:04.153 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 12:35:04.215 10084 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 12:35:04.217 10084 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146338 sec sysinv 2020-05-18 12:35:04.227 10084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:35:39.142 90717 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589804977.93-145141275236242/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589804977.93-145141275236242/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 12:35:39.145 90717 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 12:35:52.472 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-35-52 patch sysinv 2020-05-18 12:35:52.472 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-18 12:35:52.495 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_application sysinv 2020-05-18 12:35:52.498 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_controller sysinv 2020-05-18 12:35:52.646 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_worker sysinv 2020-05-18 12:35:53.091 91313 INFO sysinv.api.controllers.v1.host [-] Memory: Total=96898 MiB, Allocated=7000 MiB, 2M: 44949 pages None pages pending, 1G: 87 pages None pages pending sysinv 2020-05-18 12:35:53.127 91313 INFO sysinv.api.controllers.v1.host [-] Memory: Total=98304 MiB, Allocated=1000 MiB, 2M: 48652 pages None pages pending, 1G: 95 pages None pages pending sysinv 2020-05-18 12:35:53.138 91313 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(1): vm_mem_mib=89898, sysinv 2020-05-18 12:35:53.139 91313 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(1): {'vm_hugepages_nr_4K': 23013888} sysinv 2020-05-18 12:35:53.153 91313 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(2): vm_mem_mib=97304, sysinv 2020-05-18 12:35:53.153 91313 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(2): {'vm_hugepages_nr_4K': 24909824} sysinv 2020-05-18 12:35:53.800 91313 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query_hosts/ hdr:None payload:None sysinv 2020-05-18 12:35:54.246 91313 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'data': [{u'sw_version': u'19.12', u'hostname': u'controller-0', u'nodetype': u'controller', u'patch_failed': False, u'allow_insvc_patching': True, u'subfunctions': [u'controller', u'worker'], u'requires_reboot': False, u'installed': {}, u'state': u'idle', u'interim_state': False, u'secs_since_ack': 9, u'missing_pkgs': [], u'ip': u'192.168.204.2', u'stale_details': False, u'to_remove': [], u'patch_current': True}]} sysinv 2020-05-18 12:35:54.246 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 _semantic_mtc_check_action unlock sysinv 2020-05-18 12:35:54.247 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 action=unlock ihost_val_prenotify: {'ihost_action': 'unlock'} ihost_val: {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-18 12:35:54.247 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'unlock'} sysinv 2020-05-18 12:35:54.248 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=unlock, notify_vim=True notify_mtce=True rc=True sysinv 2020-05-18 12:35:54.248 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-18 12:35:54.249 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action unlock sysinv 2020-05-18 12:35:54.249 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_unlock_action sysinv 2020-05-18 12:35:54.249 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: unlock sysinv 2020-05-18 12:35:54.250 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-18 12:35:54.250 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-18 12:35:54.250 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-18 12:35:54.251 91313 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'unlock'} sysinv 2020-05-18 12:35:54.267 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-18 12:35:54.267 91313 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=unlock sysinv 2020-05-18 12:35:54.267 91313 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock sysinv 2020-05-18 12:35:54.268 91313 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'unlock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-18 12:35:54.269 91313 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "unlock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-18 12:35:54.274 91313 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-18 12:35:54.274 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 Perform configure_ihost. sysinv 2020-05-18 12:35:54.286 91313 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:35:54.292 90717 INFO sysinv.conductor.manager [-] configure_ihost controller-0 sysinv 2020-05-18 12:35:55.856 90717 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: None sysinv 2020-05-18 12:35:57.157 90717 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-18 12:35:57.158 90717 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-18 12:35:58.279 90717 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-18 12:35:58.326 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 Action unlock perform notify_mtce sysinv 2020-05-18 12:35:58.326 91313 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-18 12:35:58.327 91313 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "online", "invprovision": "provisioned", "peer_id": null, "administrative": "locked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "target_load": "19.12", "vsc_controllers": null, "operational": "disabled", "subfunction_avail": "online", "action": "unlock", "bm_type": "none"} sysinv 2020-05-18 12:35:58.340 91313 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-18 12:35:58.363 91313 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-35-58 patch sysinv 2020-05-18 12:35:58.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-35-58 patch sysinv 2020-05-18 12:35:58.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail', 'administrative', 'availability'] sysinv 2020-05-18 12:35:58.653 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['administrative'] sysinv 2020-05-18 12:35:58.654 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-18 12:35:58.674 91313 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-35-58 patch sysinv 2020-05-18 12:35:58.699 91313 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:36:10.433 90873 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-18 12:36:10.435 90873 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-18 12:36:10.435 90873 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-18 12:36:10.435 91313 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-18 12:36:10.453 90873 INFO oslo_service.service [-] Child 91313 exited with status 0 sysinv 2020-05-18 12:36:10.456 90873 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-18 12:36:10.456 90873 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-18 12:36:10.456 91314 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-18 12:36:10.457 90873 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-18 12:36:10.461 90873 INFO oslo_service.service [-] Child 91314 exited with status 0 sysinv 2020-05-18 12:36:11.957 90717 INFO sysinv.openstack.common.service [-] Caught SIGTERM, exiting sysinv 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 104] Connection reset by peer: error: [Errno 104] Connection reset by peer 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common Traceback (most recent call last): 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 564, in ensure 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return method(*args, **kwargs) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 644, in _consume 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return connection.drain_events(**kwargs) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 464, in drain_events 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return self.blocking_read(timeout) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 469, in blocking_read 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return self.on_inbound_frame(frame) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common callback(channel, method_sig, buf, None) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 473, in on_inbound_method 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common method_sig, payload, content, 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common listener(*args) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 593, in _on_close 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common self._x_close_ok() 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 608, in _x_close_ok 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common conn.frame_writer(1, self.channel_id, sig, args, content) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common write(view[:offset]) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 301, in write 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common self._write(s) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common tail = self.send(data, flags) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return self._send_loop(self.fd.send, data, flags) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common return send_method(data, *args) 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common error: [Errno 104] Connection reset by peer 2020-05-18 12:36:16.016 10084 ERROR sysinv.openstack.common.rpc.common sysinv 2020-05-18 12:36:16.020 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:16.030 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:36:17.031 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:17.046 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:36:20.052 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:20.076 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:36:25.078 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:25.100 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:36:32.102 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:32.118 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:36:41.119 10084 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:36:41.156 10084 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:06.626 10053 INFO sysinv.agent.lldp.manager [-] Configured sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-18 12:42:06.629 10053 INFO sysinv.agent.lldp.manager [-] Loaded sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-18 12:42:06.629 10053 INFO sysinv.agent.lldp.manager [-] Registered sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-18 12:42:06.684 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:07.685 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:07.691 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:10.695 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:10.700 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:15.704 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:15.710 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:22.712 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:22.718 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:31.723 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:31.728 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:42.735 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:42.741 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:42:55.749 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:42:55.755 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:43:10.766 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:43:10.772 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:43:27.784 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:43:27.790 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:43:46.805 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:43:46.811 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:07.822 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:07.827 10053 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.204 90794 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. sysinv 2020-05-18 12:44:26.205 90794 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". sysinv 2020-05-18 12:44:26.212 90794 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-18 12:44:26.215 90794 INFO oslo.service.wsgi [-] sysinv_api listening on 192.168.204.1:6385 sysinv 2020-05-18 12:44:26.216 90794 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-18 12:44:26.224 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.225 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.225 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.225 90794 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.226 90794 WARNING keystonemiddleware._common.config [-] The option "auth_url" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.226 90794 WARNING keystonemiddleware._common.config [-] The option "project_domain_name" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.226 90642 ERROR sysinv.common.kubernetes [-] Failed to delete custom object, Namespace kube-system: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-18 12:44:26.226 90794 WARNING keystonemiddleware._common.config [-] The option "username" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.226 90642 WARNING sysinv.conductor.kube_app [-] Failed to clear Armada locks.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-18 12:44:26.226 90794 WARNING keystonemiddleware._common.config [-] The option "project_name" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.226 90794 WARNING keystonemiddleware._common.config [-] The option "password" in conf is not known to auth_token sysinv 2020-05-18 12:44:26.228 90794 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-18 12:44:26.231 90794 INFO oslo.service.wsgi [-] sysinv_api_pxe listening on 169.254.202.1:6385 sysinv 2020-05-18 12:44:26.232 90794 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-18 12:44:26.277 90642 INFO sysinv.conductor.manager [-] sysinv-conductor start committed system={'system_mode': u'simplex', 'created_at': datetime.datetime(2020, 4, 21, 9, 57, 41, 963891, tzinfo=), 'uuid': u'1004aa49-f471-4897-97d4-8ba085382f03', 'software_version': u'19.12', 'service_project_name': u'services', 'system_type': u'All-in-one', 'name': u'3609e439-df7e-4046-bcd7-dabd81fb2f18', 'description': None, 'location': None, 'updated_at': datetime.datetime(2020, 4, 21, 9, 59, 7, 193953, tzinfo=), 'capabilities': {u'https_enabled': False, u'vswitch_type': u'none', u'region_config': False, u'sdn_enabled': False, u'shared_services': u'[]'}, 'id': 1, 'contact': None, 'security_feature': u'nopti nospectre_v2', 'services': 72, 'timezone': u'UTC', 'security_profile': u'standard', 'distributed_cloud_role': None, 'region_name': u'RegionOne'} sysinv 2020-05-18 12:44:26.302 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:26.308 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-18 12:44:26.335 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.339 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.340 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-18 12:44:26.341 90642 ERROR sysinv.conductor.manager [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)). Failed to upgrade/downgrade tiller.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-18 12:44:30.846 10053 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:30.853 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:30.861 10053 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-18 12:44:30.861 10053 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. sysinv 2020-05-18 12:44:31.564 10053 WARNING sysinv.agent.pci [-] Enabling device enp1s0f1 to query link speed sysinv 2020-05-18 12:44:31.570 10053 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp1s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:31.571 10053 WARNING sysinv.agent.pci [-] Disabling device enp1s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:31.895 10053 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f0 sysinv 2020-05-18 12:44:32.227 10053 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f1 sysinv 2020-05-18 12:44:32.550 10053 WARNING sysinv.agent.pci [-] Enabling device enp134s0f0 to query link speed sysinv 2020-05-18 12:44:32.556 10053 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f0 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:32.556 10053 WARNING sysinv.agent.pci [-] Disabling device enp134s0f0 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:32.889 10053 WARNING sysinv.agent.pci [-] Enabling device enp134s0f1 to query link speed sysinv 2020-05-18 12:44:33.079 10053 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:33.080 10053 WARNING sysinv.agent.pci [-] Disabling device enp134s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-18 12:44:33.142 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:33.163 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:44:33.184 90642 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'first_report': True} sysinv 2020-05-18 12:44:33.195 10053 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'first_report': True} sysinv 2020-05-18 12:44:33.195 10053 INFO sysinv.agent.manager [-] Agent found matching ihost: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-18 12:44:33.195 10053 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-18 12:44:33.606 90642 INFO sysinv.conductor.manager [-] port 3a2c3d57-f49c-4d28-81e8-0d4172d8024b update attr: {'speed': u'1000', 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:33.884 90642 INFO sysinv.conductor.manager [-] port 908e869d-e06a-4db2-9861-d571bdc75bc3 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:34.030 90642 INFO sysinv.conductor.manager [-] port 1fad32fc-c0e3-480d-96de-5281b9d39247 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:34.173 90642 INFO sysinv.conductor.manager [-] port 8eacf1e6-edc7-4574-ab4b-12f77b656788 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:34.446 90642 INFO sysinv.conductor.manager [-] port 525c5e74-37ac-4112-be6c-fa2f81da5b40 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:34.593 90642 INFO sysinv.conductor.manager [-] port 5af31b58-1bdf-4868-944c-1ae0f4f5cc01 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-18 12:44:34.663 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ec', 'pclass': u'Unassigned class [ff00]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'ff0000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.681 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a1d2', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.696 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'-p30', 'pdevice_id': u'a1af', 'pclass': u'USB controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0330', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.841 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1b1', 'pclass': u'Signal processing controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'118000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.859 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'mei_me', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ba', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.876 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1bb', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.892 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1be', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.910 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a182', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.931 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a1', 'pclass': u'Memory controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'058000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.948 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'i801_smbus', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a3', 'pclass': u'SMBus', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0500', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.965 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a4', 'pclass': u'Serial bus controller [0c80]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c8000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:34.985 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'2000', 'pclass': u'VGA compatible controller', 'psdevice': u'Device 86ed', 'sriov_vfs_pci_address': u'', 'pvendor': u'ASPEED Technology, Inc.', 'pvendor_id': u'1a03', 'pclass_id': u'030000', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:35.002 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'2701', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [Optane] x4 AIC (P4800X)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:35.018 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:35.036 90642 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-18 12:44:35.128 90642 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0, 1] sysinv 2020-05-18 12:44:35.128 90642 INFO sysinv.conductor.manager [-] Already in db numa_node=1 mynuma_nodes=[0, 1] sysinv 2020-05-18 12:44:35.176 90642 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:2, cores/socket=16, threads/core=2, reference:current (unchanged) sysinv 2020-05-18 12:44:35.176 90642 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 sysinv 2020-05-18 12:44:35.176 90642 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-18 12:44:35.177 90642 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 sysinv 2020-05-18 12:44:35.177 90642 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-18 12:44:37.108 90642 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': 0, 'name': u'enp1s0f0'}] sysinv 2020-05-18 12:44:37.114 10053 INFO sysinv.agent.manager [-] Affine platform interface enp1s0f0 with cpulist 0,1 sysinv 2020-05-18 12:44:37.222 10053 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp1s0f0 0,1 return 0 sysinv 2020-05-18 12:44:37.248 10053 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:aceda3e9b5e8 sysinv 2020-05-18 12:44:37.253 90642 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'config_applied': u'f462e90a-7bd6-4568-8a1d-02d27a6f2c52', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-18 12:44:37.329 90642 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config f462e90a-7bd6-4568-8a1d-02d27a6f2c52 fmAPI.cpp(137): Connected to FM Manager. sysinv 2020-05-18 12:44:37.369 10053 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': 'f462e90a-7bd6-4568-8a1d-02d27a6f2c52', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-18 12:44:40.050 10053 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:5e:00.0-nvme-1: 698 ... large disk defaults sysinv 2020-05-18 12:44:40.074 90642 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-05-18 12:44:40.074 90642 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-05-18 12:44:40.075 90642 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-05-18 12:44:40.075 90642 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-05-18 12:44:40.080 10053 INFO sysinv.agent.manager [-] Initial inventory completed host 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-18 12:44:41.067 91235 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. sysinv 2020-05-18 12:44:42.373 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:44:42.497 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-44-42 patch sysinv 2020-05-18 12:44:42.497 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:44:44.075 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-44-44 patch sysinv 2020-05-18 12:44:44.076 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-18 12:44:44.095 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-44-44 patch sysinv 2020-05-18 12:44:44.148 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:44:59.247 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:44:59.314 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:44:59.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:44:59.809 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-44-59 patch sysinv 2020-05-18 12:44:59.809 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-18 12:44:59.887 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-44-59 patch sysinv 2020-05-18 12:44:59.932 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:45:26.692 90642 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '64f473fa-1ab0-4079-b619-fdb925df87e9'. sysinv 2020-05-18 12:45:26.704 90642 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 64f473fa-1ab0-4079-b619-fdb925df87e9, applied f462e90a-7bd6-4568-8a1d-02d27a6f2c52 sysinv 2020-05-18 12:45:26.704 90642 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: f462e90a-7bd6-4568-8a1d-02d27a6f2c52 vs. target: 64f473fa-1ab0-4079-b619-fdb925df87e9. sysinv 2020-05-18 12:45:26.760 90642 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-18 12:45:26.760 90642 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=64f473fa-1ab0-4079-b619-fdb925df87e9, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'] sysinv 2020-05-18 12:45:26.768 90642 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-18 12:45:27.970 90642 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-18 12:45:27.971 90642 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-18 12:45:29.771 90642 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-18 12:45:29.774 90642 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-18 12:45:29.774 90642 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 64f473fa-1ab0-4079-b619-fdb925df87e9 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} to agent sysinv 2020-05-18 12:45:29.777 10053 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 64f473fa-1ab0-4079-b619-fdb925df87e9 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} controller sysinv 2020-05-18 12:45:29.777 10053 INFO sysinv.agent.manager [-] controller-active sysinv 2020-05-18 12:45:29.777 10053 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-05-18 12:45:29.844 90642 INFO sysinv.conductor.manager [-] drbd-overview: pgsql-20.0, platform-9.8, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribution-16.0 sysinv 2020-05-18 12:45:29.844 90642 INFO sysinv.conductor.manager [-] lvdisplay: pgsql-20.0, platform-10.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribution-16.0 sysinv 2020-05-18 12:45:29.855 90642 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config f462e90a-7bd6-4568-8a1d-02d27a6f2c52 sysinv 2020-05-18 12:45:46.757 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-18 12:45:46.877 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-45-46 patch sysinv 2020-05-18 12:45:46.877 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 12:45:46.927 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-45-46 patch sysinv 2020-05-18 12:45:46.928 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-18 12:45:46.990 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-45-46 patch sysinv 2020-05-18 12:46:16.805 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-46-16 patch sysinv 2020-05-18 12:46:16.806 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-18 12:46:16.966 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-46-16 patch sysinv 2020-05-18 12:46:33.492 10053 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime']. sysinv 2020-05-18 12:46:33.493 10053 INFO sysinv.agent.manager [-] Agent config applied 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-18 12:46:33.523 90642 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-18 12:46:56.806 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-46-56 patch sysinv 2020-05-18 12:46:56.806 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-18 12:46:56.867 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-46-56 patch sysinv 2020-05-18 12:47:52.849 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-47-52 patch sysinv 2020-05-18 12:47:52.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-18 12:47:52.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-enabled sysinv 2020-05-18 12:47:52.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-enabled ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-enabled, notify_vim=False notify_mtce=False rc=True sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-enabled sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 received services-enabled task= vim_progress_status=services-disabled sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-enabled sysinv 2020-05-18 12:47:52.851 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-18 12:47:52.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-18 12:47:52.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-18 12:47:52.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-enabled'} sysinv 2020-05-18 12:47:52.916 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 notify_availability=services-enabled sysinv 2020-05-18 12:47:52.952 91235 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 12:47:52.956 90642 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'availability': u'services-enabled'} sysinv 2020-05-18 12:47:53.007 90642 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-18 12:47:53.013 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-18-12-47-53 patch sysinv 2020-05-18 12:55:46.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-12-55-46 patch sysinv 2020-05-18 12:55:46.969 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:00:47.001 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-00-46 patch sysinv 2020-05-18 13:00:47.001 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:05:47.033 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-05-47 patch sysinv 2020-05-18 13:05:47.034 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:10:47.064 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-10-47 patch sysinv 2020-05-18 13:10:47.065 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:15:47.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-15-47 patch sysinv 2020-05-18 13:15:47.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:20:47.206 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-20-47 patch sysinv 2020-05-18 13:20:47.207 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:25:47.158 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-25-47 patch sysinv 2020-05-18 13:25:47.159 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:30:47.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-30-47 patch sysinv 2020-05-18 13:30:47.196 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:35:47.374 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-35-47 patch sysinv 2020-05-18 13:35:47.375 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:40:47.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-40-47 patch sysinv 2020-05-18 13:40:47.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:44:26.390 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-18 13:44:26.399 90642 INFO sysinv.conductor.manager [-] Running tiller image: registry.local:9001/gcr.io/kubernetes-helm/tiller:v2.13.1 sysinv 2020-05-18 13:44:26.399 90642 INFO sysinv.conductor.manager [-] Requested tiller version: v2.13.1 sysinv 2020-05-18 13:44:26.400 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 13:45:36.771 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 13:45:36.780 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:45:47.323 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-45-47 patch sysinv 2020-05-18 13:45:47.323 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:46:36.841 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 13:46:36.844 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140698 sec sysinv 2020-05-18 13:46:36.853 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:47:36.927 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 13:47:36.936 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:48:37.002 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 13:48:37.004 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.159167 sec sysinv 2020-05-18 13:48:37.031 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:49:37.101 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 13:49:37.114 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 13:50:37.179 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 13:50:37.180 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.17625 sec sysinv 2020-05-18 13:50:37.189 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:50:47.424 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-50-47 patch sysinv 2020-05-18 13:50:47.425 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 13:51:06.960 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589809505.55-19783257067140/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589809505.55-19783257067140/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 13:51:06.962 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 13:51:07.242 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:51:07.316 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:51:07.327 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:51:07.328 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 13:55:47.399 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-13-55-47 patch sysinv 2020-05-18 13:55:47.400 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:00:47.469 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-00-47 patch sysinv 2020-05-18 14:00:47.469 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:05:47.457 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-05-47 patch sysinv 2020-05-18 14:05:47.458 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:10:47.564 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-10-47 patch sysinv 2020-05-18 14:10:47.565 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:15:47.652 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-15-47 patch sysinv 2020-05-18 14:15:47.652 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:20:47.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-20-47 patch sysinv 2020-05-18 14:20:47.591 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:25:47.527 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-25-47 patch sysinv 2020-05-18 14:25:47.528 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:30:47.683 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-30-47 patch sysinv 2020-05-18 14:30:47.684 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:35:47.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-35-47 patch sysinv 2020-05-18 14:35:47.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:40:47.748 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-40-47 patch sysinv 2020-05-18 14:40:47.749 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:45:47.731 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-45-47 patch sysinv 2020-05-18 14:45:47.732 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:50:47.815 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-50-47 patch sysinv 2020-05-18 14:50:47.816 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:51:06.987 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 14:52:40.682 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 14:52:40.698 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:53:40.760 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 14:53:40.761 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146829 sec sysinv 2020-05-18 14:53:40.770 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:54:40.832 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 14:54:40.840 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:55:40.901 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 14:55:40.903 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14125 sec sysinv 2020-05-18 14:55:40.913 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:55:47.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-14-55-47 patch sysinv 2020-05-18 14:55:47.772 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 14:56:40.974 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 14:56:40.983 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 14:57:41.044 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 14:57:41.045 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141436 sec sysinv 2020-05-18 14:57:41.053 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 14:57:45.332 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589813504.17-277317184724781/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589813504.17-277317184724781/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 14:57:45.333 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 15:00:47.860 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-00-47 patch sysinv 2020-05-18 15:00:47.861 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:05:47.849 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-05-47 patch sysinv 2020-05-18 15:05:47.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:10:47.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-10-47 patch sysinv 2020-05-18 15:10:47.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:15:47.926 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-15-47 patch sysinv 2020-05-18 15:15:47.927 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:20:48.084 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-20-48 patch sysinv 2020-05-18 15:20:48.084 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:25:47.979 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-25-47 patch sysinv 2020-05-18 15:25:47.979 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:30:47.983 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-30-47 patch sysinv 2020-05-18 15:30:47.983 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:35:48.135 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-35-48 patch sysinv 2020-05-18 15:35:48.135 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:40:48.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-40-47 patch sysinv 2020-05-18 15:40:48.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:45:48.086 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-45-48 patch sysinv 2020-05-18 15:45:48.087 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:50:48.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-50-48 patch sysinv 2020-05-18 15:50:48.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:55:48.208 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-15-55-48 patch sysinv 2020-05-18 15:55:48.208 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 15:57:45.334 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "idisk_update_by_ihost" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "idisk_update_by_ihost" info: "" 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1257, in agent_audit 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task idisk) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 354, in idisk_update_by_ihost 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task idisk_dict_array=idisk_dict_array)) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "idisk_update_by_ihost" info: "" 2020-05-18 15:58:45.719 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 15:58:45.720 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 1.267506 sec sysinv 2020-05-18 15:58:45.729 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 15:59:45.799 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 15:59:45.807 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 16:00:45.868 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 16:00:45.869 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148783 sec sysinv 2020-05-18 16:00:45.878 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 16:00:48.194 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-00-48 patch sysinv 2020-05-18 16:00:48.194 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:01:45.940 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 16:01:45.950 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 16:02:46.010 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 16:02:46.011 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14175 sec sysinv 2020-05-18 16:02:46.022 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 16:03:46.082 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 16:03:46.094 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 16:04:23.900 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589817502.8-158274432672258/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589817502.8-158274432672258/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 16:04:23.902 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 16:04:26.955 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 40.943574 sec sysinv 2020-05-18 16:05:48.264 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-05-48 patch sysinv 2020-05-18 16:05:48.264 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:10:48.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-10-48 patch sysinv 2020-05-18 16:10:48.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:15:48.347 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-15-48 patch sysinv 2020-05-18 16:15:48.347 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:20:48.332 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-20-48 patch sysinv 2020-05-18 16:20:48.333 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:25:48.310 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-25-48 patch sysinv 2020-05-18 16:25:48.311 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:30:48.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-30-48 patch sysinv 2020-05-18 16:30:48.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:35:48.385 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-35-48 patch sysinv 2020-05-18 16:35:48.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:40:48.765 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-40-48 patch sysinv 2020-05-18 16:40:48.766 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:45:48.438 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-45-48 patch sysinv 2020-05-18 16:45:48.438 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:50:48.516 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-50-48 patch sysinv 2020-05-18 16:50:48.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 16:55:48.554 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-16-55-48 patch sysinv 2020-05-18 16:55:48.554 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:00:48.555 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-00-48 patch sysinv 2020-05-18 17:00:48.555 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:04:23.954 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 17:05:30.405 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 17:05:30.414 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:05:48.665 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-05-48 patch sysinv 2020-05-18 17:05:48.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:06:30.474 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 17:06:30.475 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142297 sec sysinv 2020-05-18 17:06:30.488 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:07:30.549 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 17:07:30.558 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:08:30.619 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 17:08:30.620 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144506 sec sysinv 2020-05-18 17:08:30.630 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:09:30.693 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 17:09:30.735 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 17:10:30.802 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 17:10:30.803 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.182683 sec sysinv 2020-05-18 17:10:30.825 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:10:48.620 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-10-48 patch sysinv 2020-05-18 17:10:48.620 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:11:03.208 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589821501.91-146554718021745/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589821501.91-146554718021745/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 17:11:03.209 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 17:11:03.384 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 17:15:48.665 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-15-48 patch sysinv 2020-05-18 17:15:48.665 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:20:48.694 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-20-48 patch sysinv 2020-05-18 17:20:48.695 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:25:48.704 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-25-48 patch sysinv 2020-05-18 17:25:48.704 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:30:48.754 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-30-48 patch sysinv 2020-05-18 17:30:48.755 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:35:48.838 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-35-48 patch sysinv 2020-05-18 17:35:48.839 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:40:49.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-40-49 patch sysinv 2020-05-18 17:40:49.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:45:48.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-45-48 patch sysinv 2020-05-18 17:45:48.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:50:48.870 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-50-48 patch sysinv 2020-05-18 17:50:48.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 17:55:48.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-17-55-48 patch sysinv 2020-05-18 17:55:48.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:00:48.922 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-00-48 patch sysinv 2020-05-18 18:00:48.923 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:05:48.861 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-05-48 patch sysinv 2020-05-18 18:05:48.862 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:10:49.039 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-10-48 patch sysinv 2020-05-18 18:10:49.040 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:11:03.236 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 18:12:34.295 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 18:12:34.306 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:13:34.367 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 18:13:34.368 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139848 sec sysinv 2020-05-18 18:13:34.377 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:14:34.438 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 18:14:34.447 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:15:34.508 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 18:15:34.510 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140947 sec sysinv 2020-05-18 18:15:34.518 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:15:48.989 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-15-48 patch sysinv 2020-05-18 18:15:48.989 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:16:34.583 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 18:16:34.591 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 18:17:34.653 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 18:17:34.654 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143813 sec sysinv 2020-05-18 18:17:34.664 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 18:17:41.640 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589825500.5-43058558247388/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589825500.5-43058558247388/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 18:17:41.641 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 18:20:48.994 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-20-48 patch sysinv 2020-05-18 18:20:48.994 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:25:49.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-25-49 patch sysinv 2020-05-18 18:25:49.101 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:30:49.010 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-30-48 patch sysinv 2020-05-18 18:30:49.011 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:35:49.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-35-49 patch sysinv 2020-05-18 18:35:49.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:40:49.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-40-49 patch sysinv 2020-05-18 18:40:49.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:45:49.209 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-45-49 patch sysinv 2020-05-18 18:45:49.210 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:50:49.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-50-49 patch sysinv 2020-05-18 18:50:49.250 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 18:55:49.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-18-55-49 patch sysinv 2020-05-18 18:55:49.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:00:49.328 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-00-49 patch sysinv 2020-05-18 19:00:49.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:05:49.327 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-05-49 patch sysinv 2020-05-18 19:05:49.328 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:10:49.353 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-10-49 patch sysinv 2020-05-18 19:10:49.354 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:15:49.421 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-15-49 patch sysinv 2020-05-18 19:15:49.422 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:17:41.643 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 19:19:38.203 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 19:19:38.211 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 19:20:38.273 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 19:20:38.274 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.135736 sec sysinv 2020-05-18 19:20:38.282 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 19:20:49.461 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-20-49 patch sysinv 2020-05-18 19:20:49.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:21:38.344 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 19:21:38.355 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 19:22:38.428 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 19:22:38.429 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154952 sec sysinv 2020-05-18 19:22:38.442 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 19:23:38.503 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 19:23:38.512 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 19:24:23.949 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589829502.85-121742389188828/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589829502.85-121742389188828/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 19:24:23.950 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 19:24:26.303 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 47.873556 sec sysinv 2020-05-18 19:25:49.478 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-25-49 patch sysinv 2020-05-18 19:25:49.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:30:49.499 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-30-49 patch sysinv 2020-05-18 19:30:49.500 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:35:49.597 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-35-49 patch sysinv 2020-05-18 19:35:49.598 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:40:49.920 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-40-49 patch sysinv 2020-05-18 19:40:49.921 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:45:49.634 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-45-49 patch sysinv 2020-05-18 19:45:49.635 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:50:49.570 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-50-49 patch sysinv 2020-05-18 19:50:49.571 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 19:55:49.648 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-19-55-49 patch sysinv 2020-05-18 19:55:49.649 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:00:49.621 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-00-49 patch sysinv 2020-05-18 20:00:49.622 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:05:49.712 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-05-49 patch sysinv 2020-05-18 20:05:49.712 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:10:49.744 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-10-49 patch sysinv 2020-05-18 20:10:49.745 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:15:49.841 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-15-49 patch sysinv 2020-05-18 20:15:49.842 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:20:49.782 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-20-49 patch sysinv 2020-05-18 20:20:49.782 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:24:24.004 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 20:25:29.745 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 20:25:29.754 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:25:49.894 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-25-49 patch sysinv 2020-05-18 20:25:49.895 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:26:29.817 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 20:26:29.818 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140623 sec sysinv 2020-05-18 20:26:29.826 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:27:29.889 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 20:27:29.898 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:28:29.961 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 20:28:29.962 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143986 sec sysinv 2020-05-18 20:28:29.972 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:29:30.034 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 20:29:30.044 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 20:30:30.102 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 20:30:30.103 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140599 sec sysinv 2020-05-18 20:30:30.115 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 20:30:49.876 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-30-49 patch sysinv 2020-05-18 20:30:49.877 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:31:03.193 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589833501.96-73511750619221/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589833501.96-73511750619221/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 20:31:03.194 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 20:35:49.918 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-35-49 patch sysinv 2020-05-18 20:35:49.919 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:40:49.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-40-49 patch sysinv 2020-05-18 20:40:49.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:45:49.865 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-45-49 patch sysinv 2020-05-18 20:45:49.866 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:50:50.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-50-49 patch sysinv 2020-05-18 20:50:50.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 20:55:50.013 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-20-55-49 patch sysinv 2020-05-18 20:55:50.013 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:00:50.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-00-50 patch sysinv 2020-05-18 21:00:50.119 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:05:50.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-05-49 patch sysinv 2020-05-18 21:05:50.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:10:50.194 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-10-50 patch sysinv 2020-05-18 21:10:50.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:15:50.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-15-50 patch sysinv 2020-05-18 21:15:50.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:20:50.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-20-50 patch sysinv 2020-05-18 21:20:50.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:25:50.158 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-25-50 patch sysinv 2020-05-18 21:25:50.159 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:30:50.360 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-30-50 patch sysinv 2020-05-18 21:30:50.361 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:31:03.222 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 21:32:33.603 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 21:32:33.613 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:33:33.699 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 21:33:33.700 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.164763 sec sysinv 2020-05-18 21:33:33.715 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:34:33.784 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 21:34:33.794 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:35:33.855 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 21:35:33.856 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.155771 sec sysinv 2020-05-18 21:35:33.864 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:35:50.208 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-35-50 patch sysinv 2020-05-18 21:35:50.208 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:36:33.926 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 21:36:33.947 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 21:37:34.008 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 21:37:34.009 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152675 sec sysinv 2020-05-18 21:37:34.018 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:37:42.286 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589837501.23-228204043133922/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589837501.23-228204043133922/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 21:37:42.287 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 21:37:42.612 90642 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 21:40:50.242 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-40-50 patch sysinv 2020-05-18 21:40:50.243 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:45:50.255 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-45-50 patch sysinv 2020-05-18 21:45:50.255 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:50:50.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-50-50 patch sysinv 2020-05-18 21:50:50.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 21:55:50.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-21-55-50 patch sysinv 2020-05-18 21:55:50.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:00:50.392 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-00-50 patch sysinv 2020-05-18 22:00:50.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:05:50.444 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-05-50 patch sysinv 2020-05-18 22:05:50.445 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:10:50.573 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-10-50 patch sysinv 2020-05-18 22:10:50.574 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:15:50.534 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-15-50 patch sysinv 2020-05-18 22:15:50.535 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:20:50.472 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-20-50 patch sysinv 2020-05-18 22:20:50.472 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:25:50.559 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-25-50 patch sysinv 2020-05-18 22:25:50.559 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:30:50.541 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-30-50 patch sysinv 2020-05-18 22:30:50.542 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:35:50.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-35-50 patch sysinv 2020-05-18 22:35:50.663 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:37:42.291 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 22:39:37.546 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 22:39:37.558 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 22:40:37.633 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 22:40:37.634 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.155362 sec sysinv 2020-05-18 22:40:37.643 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 22:40:50.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-40-50 patch sysinv 2020-05-18 22:40:50.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:41:37.704 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 22:41:37.714 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 22:42:37.779 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 22:42:37.780 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146057 sec sysinv 2020-05-18 22:42:37.788 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 22:43:37.850 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 22:43:37.857 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 22:44:22.740 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589841501.55-175354734403887/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589841501.55-175354734403887/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 22:44:22.741 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 22:44:25.402 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 47.621461 sec sysinv 2020-05-18 22:45:50.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-45-50 patch sysinv 2020-05-18 22:45:50.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:50:50.766 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-50-50 patch sysinv 2020-05-18 22:50:50.767 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 22:55:50.819 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-22-55-50 patch sysinv 2020-05-18 22:55:50.819 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:00:50.754 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-00-50 patch sysinv 2020-05-18 23:00:50.755 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:05:50.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-05-50 patch sysinv 2020-05-18 23:05:50.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:10:50.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-10-50 patch sysinv 2020-05-18 23:10:50.774 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:15:50.895 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-15-50 patch sysinv 2020-05-18 23:15:50.896 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:20:50.841 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-20-50 patch sysinv 2020-05-18 23:20:50.841 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:25:50.948 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-25-50 patch sysinv 2020-05-18 23:25:50.948 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:30:51.007 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-30-50 patch sysinv 2020-05-18 23:30:51.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:35:51.093 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-35-51 patch sysinv 2020-05-18 23:35:51.093 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:40:50.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-40-50 patch sysinv 2020-05-18 23:40:50.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:44:22.793 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-18 23:45:28.846 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 23:45:28.858 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:45:51.032 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-45-51 patch sysinv 2020-05-18 23:45:51.032 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:46:28.921 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 23:46:28.922 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145356 sec sysinv 2020-05-18 23:46:28.937 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:47:28.998 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 23:47:29.008 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:48:29.083 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 23:48:29.084 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.161766 sec sysinv 2020-05-18 23:48:29.102 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:49:29.162 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-18 23:49:29.170 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-18 23:50:29.230 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-18 23:50:29.231 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146978 sec sysinv 2020-05-18 23:50:29.240 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-18 23:50:50.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-50-50 patch sysinv 2020-05-18 23:50:50.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-18 23:51:01.375 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589845500.18-226512973811926/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589845500.18-226512973811926/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-18 23:51:01.376 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-18 23:55:50.998 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-18-23-55-50 patch sysinv 2020-05-18 23:55:50.999 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:00:51.033 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-00-51 patch sysinv 2020-05-19 00:00:51.034 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:05:51.057 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-05-51 patch sysinv 2020-05-19 00:05:51.057 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:10:51.127 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-10-51 patch sysinv 2020-05-19 00:10:51.127 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:15:51.306 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-15-51 patch sysinv 2020-05-19 00:15:51.306 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:20:51.201 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-20-51 patch sysinv 2020-05-19 00:20:51.201 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:25:51.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-25-51 patch sysinv 2020-05-19 00:25:51.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:30:51.338 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-30-51 patch sysinv 2020-05-19 00:30:51.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:35:51.330 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-35-51 patch sysinv 2020-05-19 00:35:51.330 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:40:51.699 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-40-51 patch sysinv 2020-05-19 00:40:51.700 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:45:51.380 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-45-51 patch sysinv 2020-05-19 00:45:51.381 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:50:51.399 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-50-51 patch sysinv 2020-05-19 00:50:51.399 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:51:01.404 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 00:52:32.716 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 00:52:32.724 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:53:32.784 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 00:53:32.786 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.13464 sec sysinv 2020-05-19 00:53:32.804 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:54:32.865 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 00:54:32.873 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:55:32.940 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 00:55:32.941 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154971 sec sysinv 2020-05-19 00:55:32.955 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:55:51.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-00-55-51 patch sysinv 2020-05-19 00:55:51.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 00:56:33.018 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 00:56:33.026 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 00:57:33.099 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 00:57:33.100 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15852 sec sysinv 2020-05-19 00:57:33.113 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 00:57:41.408 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589849500.24-7167423888806/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589849500.24-7167423888806/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 00:57:41.409 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 01:00:51.432 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-00-51 patch sysinv 2020-05-19 01:00:51.432 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:05:51.545 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-05-51 patch sysinv 2020-05-19 01:05:51.545 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:10:51.550 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-10-51 patch sysinv 2020-05-19 01:10:51.551 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:15:51.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-15-51 patch sysinv 2020-05-19 01:15:51.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:20:51.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-20-51 patch sysinv 2020-05-19 01:20:51.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:25:51.620 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-25-51 patch sysinv 2020-05-19 01:25:51.621 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:30:51.713 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-30-51 patch sysinv 2020-05-19 01:30:51.713 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:35:51.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-35-51 patch sysinv 2020-05-19 01:35:51.772 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:40:52.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-40-52 patch sysinv 2020-05-19 01:40:52.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:45:51.710 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-45-51 patch sysinv 2020-05-19 01:45:51.711 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:50:51.723 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-50-51 patch sysinv 2020-05-19 01:50:51.724 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:55:51.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-01-55-51 patch sysinv 2020-05-19 01:55:51.960 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 01:57:41.412 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 01:59:36.621 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 01:59:36.630 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 02:00:36.691 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 02:00:36.692 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139364 sec sysinv 2020-05-19 02:00:36.701 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 02:00:51.831 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-00-51 patch sysinv 2020-05-19 02:00:51.832 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:01:36.768 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 02:01:36.776 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 02:02:36.838 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 02:02:36.839 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146603 sec sysinv 2020-05-19 02:02:36.855 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 02:03:36.913 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 02:03:36.921 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 02:04:22.277 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589853501.18-88294478732091/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589853501.18-88294478732091/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 02:04:22.279 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 02:04:24.907 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 48.067458 sec sysinv 2020-05-19 02:05:51.843 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-05-51 patch sysinv 2020-05-19 02:05:51.843 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:10:51.910 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-10-51 patch sysinv 2020-05-19 02:10:51.911 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:15:51.936 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-15-51 patch sysinv 2020-05-19 02:15:51.937 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:20:51.984 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-20-51 patch sysinv 2020-05-19 02:20:51.985 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:25:51.960 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-25-51 patch sysinv 2020-05-19 02:25:51.960 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:30:52.048 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-30-52 patch sysinv 2020-05-19 02:30:52.049 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:35:51.974 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-35-51 patch sysinv 2020-05-19 02:35:51.975 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:40:51.995 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-40-51 patch sysinv 2020-05-19 02:40:51.995 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:45:52.110 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-45-52 patch sysinv 2020-05-19 02:45:52.111 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:50:52.203 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-50-52 patch sysinv 2020-05-19 02:50:52.203 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 02:55:52.226 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-02-55-52 patch sysinv 2020-05-19 02:55:52.227 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:00:52.200 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-00-52 patch sysinv 2020-05-19 03:00:52.200 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:04:22.331 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 03:05:28.354 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 03:05:28.374 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:05:52.149 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-05-52 patch sysinv 2020-05-19 03:05:52.150 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:06:28.437 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 03:06:28.438 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152407 sec sysinv 2020-05-19 03:06:28.450 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:07:28.514 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 03:07:28.533 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:08:28.603 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 03:08:28.605 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.166412 sec sysinv 2020-05-19 03:08:28.621 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:09:28.693 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 03:09:28.705 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 03:10:28.768 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 03:10:28.769 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.163713 sec sysinv 2020-05-19 03:10:28.785 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 03:10:52.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-10-52 patch sysinv 2020-05-19 03:10:52.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:11:03.406 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589857502.27-206027880930055/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589857502.27-206027880930055/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 03:11:03.407 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 03:15:52.347 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-15-52 patch sysinv 2020-05-19 03:15:52.348 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:20:52.343 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-20-52 patch sysinv 2020-05-19 03:20:52.344 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:25:52.263 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-25-52 patch sysinv 2020-05-19 03:25:52.264 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:30:52.411 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-30-52 patch sysinv 2020-05-19 03:30:52.411 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:35:52.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-35-52 patch sysinv 2020-05-19 03:35:52.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:40:52.818 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-40-52 patch sysinv 2020-05-19 03:40:52.819 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:45:52.511 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-45-52 patch sysinv 2020-05-19 03:45:52.512 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:50:52.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-50-52 patch sysinv 2020-05-19 03:50:52.482 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 03:55:52.610 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-03-55-52 patch sysinv 2020-05-19 03:55:52.610 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:00:52.582 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-00-52 patch sysinv 2020-05-19 04:00:52.582 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:05:52.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-05-52 patch sysinv 2020-05-19 04:05:52.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:10:52.584 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-10-52 patch sysinv 2020-05-19 04:10:52.584 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:11:03.436 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 04:12:32.263 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 04:12:32.272 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:13:32.334 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 04:13:32.335 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142692 sec sysinv 2020-05-19 04:13:32.344 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:14:32.405 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 04:14:32.413 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:15:32.474 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 04:15:32.475 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14009 sec sysinv 2020-05-19 04:15:32.490 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:15:52.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-15-52 patch sysinv 2020-05-19 04:15:52.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:16:32.551 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 04:16:32.560 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 04:17:32.620 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 04:17:32.621 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145523 sec sysinv 2020-05-19 04:17:32.630 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 04:17:43.358 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589861502.27-246834246776399/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589861502.27-246834246776399/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 04:17:43.359 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 04:20:52.632 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-20-52 patch sysinv 2020-05-19 04:20:52.633 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:25:52.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-25-52 patch sysinv 2020-05-19 04:25:52.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:30:52.727 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-30-52 patch sysinv 2020-05-19 04:30:52.727 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:35:52.885 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-35-52 patch sysinv 2020-05-19 04:35:52.885 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:40:53.203 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-40-53 patch sysinv 2020-05-19 04:40:53.203 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:45:52.801 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-45-52 patch sysinv 2020-05-19 04:45:52.802 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:50:52.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-50-52 patch sysinv 2020-05-19 04:50:52.872 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 04:55:52.889 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-04-55-52 patch sysinv 2020-05-19 04:55:52.889 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:00:52.940 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-00-52 patch sysinv 2020-05-19 05:00:52.940 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:05:52.961 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-05-52 patch sysinv 2020-05-19 05:05:52.961 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:10:53.005 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-10-52 patch sysinv 2020-05-19 05:10:53.005 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:15:53.003 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-15-52 patch sysinv 2020-05-19 05:15:53.004 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:17:43.365 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 05:19:36.166 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 05:19:36.178 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 05:20:36.241 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 05:20:36.242 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146852 sec sysinv 2020-05-19 05:20:36.252 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 05:20:53.142 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-20-53 patch sysinv 2020-05-19 05:20:53.143 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:21:36.313 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 05:21:36.322 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 05:22:36.384 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 05:22:36.385 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142361 sec sysinv 2020-05-19 05:22:36.395 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 05:23:36.457 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 05:23:36.476 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 05:24:22.569 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589865501.52-52056115480065/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589865501.52-52056115480065/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 05:24:22.570 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 05:24:24.662 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 48.276373 sec sysinv 2020-05-19 05:25:53.075 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-25-53 patch sysinv 2020-05-19 05:25:53.076 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:30:53.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-30-53 patch sysinv 2020-05-19 05:30:53.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:35:53.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-35-53 patch sysinv 2020-05-19 05:35:53.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:40:53.637 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-40-53 patch sysinv 2020-05-19 05:40:53.639 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:45:53.222 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-45-53 patch sysinv 2020-05-19 05:45:53.223 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:50:53.187 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-50-53 patch sysinv 2020-05-19 05:50:53.188 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 05:55:53.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-05-55-53 patch sysinv 2020-05-19 05:55:53.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:00:53.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-00-53 patch sysinv 2020-05-19 06:00:53.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:05:53.251 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-05-53 patch sysinv 2020-05-19 06:05:53.252 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:10:53.452 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-10-53 patch sysinv 2020-05-19 06:10:53.453 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:15:53.461 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-15-53 patch sysinv 2020-05-19 06:15:53.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:20:53.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-20-53 patch sysinv 2020-05-19 06:20:53.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:24:22.624 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 06:25:28.110 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 06:25:28.120 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:25:53.376 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-25-53 patch sysinv 2020-05-19 06:25:53.377 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:26:28.184 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 06:26:28.185 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143797 sec sysinv 2020-05-19 06:26:28.195 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:27:28.256 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 06:27:28.268 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:28:28.331 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 06:28:28.333 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14718 sec sysinv 2020-05-19 06:28:28.347 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:29:28.410 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 06:29:28.423 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 06:30:28.484 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 06:30:28.485 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152415 sec sysinv 2020-05-19 06:30:28.497 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 06:30:53.463 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-30-53 patch sysinv 2020-05-19 06:30:53.464 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:31:04.220 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589869502.74-125481101307914/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589869502.74-125481101307914/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 06:31:04.221 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 06:35:53.580 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-35-53 patch sysinv 2020-05-19 06:35:53.581 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:40:53.544 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-40-53 patch sysinv 2020-05-19 06:40:53.546 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:45:53.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-45-53 patch sysinv 2020-05-19 06:45:53.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:50:53.570 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-50-53 patch sysinv 2020-05-19 06:50:53.571 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 06:55:53.685 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-06-55-53 patch sysinv 2020-05-19 06:55:53.685 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:00:53.767 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-00-53 patch sysinv 2020-05-19 07:00:53.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:05:53.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-05-53 patch sysinv 2020-05-19 07:05:53.742 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:10:53.750 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-10-53 patch sysinv 2020-05-19 07:10:53.751 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:15:53.825 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-15-53 patch sysinv 2020-05-19 07:15:53.826 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:20:53.812 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-20-53 patch sysinv 2020-05-19 07:20:53.813 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:25:53.866 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-25-53 patch sysinv 2020-05-19 07:25:53.867 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:30:53.859 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-30-53 patch sysinv 2020-05-19 07:30:53.859 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:31:04.253 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 07:32:31.974 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 07:32:31.987 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:33:32.050 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 07:33:32.051 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145395 sec sysinv 2020-05-19 07:33:32.063 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:34:32.124 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 07:34:32.134 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:35:32.196 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 07:35:32.197 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14574 sec sysinv 2020-05-19 07:35:32.206 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:35:53.797 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-35-53 patch sysinv 2020-05-19 07:35:53.798 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:36:32.279 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 07:36:32.295 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 07:37:32.358 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 07:37:32.359 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.161405 sec sysinv 2020-05-19 07:37:32.368 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 07:37:46.945 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589873505.76-272048236341765/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589873505.76-272048236341765/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 07:37:46.946 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 07:40:53.912 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-40-53 patch sysinv 2020-05-19 07:40:53.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:45:53.986 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-45-53 patch sysinv 2020-05-19 07:45:53.987 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:50:54.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-50-54 patch sysinv 2020-05-19 07:50:54.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 07:55:53.926 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-07-55-53 patch sysinv 2020-05-19 07:55:53.926 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:00:54.186 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-00-54 patch sysinv 2020-05-19 08:00:54.187 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:05:54.059 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-05-54 patch sysinv 2020-05-19 08:05:54.059 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:10:54.022 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-10-53 patch sysinv 2020-05-19 08:10:54.023 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:15:54.043 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-15-54 patch sysinv 2020-05-19 08:15:54.043 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:20:54.184 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-20-54 patch sysinv 2020-05-19 08:20:54.185 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:22:46.258 90642 INFO sysinv.conductor.manager [-] Updating parameters configuration for service: docker sysinv 2020-05-19 08:22:46.272 90642 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '44741f43-5481-498d-9af4-668dfdf58f3c'. sysinv 2020-05-19 08:22:46.282 90642 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 44741f43-5481-498d-9af4-668dfdf58f3c, applied 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-19 08:22:46.283 90642 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 64f473fa-1ab0-4079-b619-fdb925df87e9 vs. target: 44741f43-5481-498d-9af4-668dfdf58f3c. sysinv 2020-05-19 08:22:46.293 90642 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=44741f43-5481-498d-9af4-668dfdf58f3c sysinv 2020-05-19 08:25:54.251 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-25-54 patch sysinv 2020-05-19 08:25:54.252 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:30:54.330 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-30-54 patch sysinv 2020-05-19 08:30:54.330 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:35:54.373 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-35-54 patch sysinv 2020-05-19 08:35:54.373 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:37:46.956 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 08:39:35.895 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 08:39:35.907 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 08:40:35.970 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 08:40:35.971 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14295 sec sysinv 2020-05-19 08:40:35.983 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 08:40:54.683 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-40-54 patch sysinv 2020-05-19 08:40:54.684 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:41:36.044 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 08:41:36.052 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 08:42:36.115 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 08:42:36.116 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144699 sec sysinv 2020-05-19 08:42:36.127 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 08:43:36.188 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 08:43:36.197 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 08:44:27.439 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589877506.3-111691461302775/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589877506.3-111691461302775/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 08:44:27.441 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 08:44:29.830 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 53.714184 sec sysinv 2020-05-19 08:45:54.291 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-45-54 patch sysinv 2020-05-19 08:45:54.292 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:50:54.364 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-50-54 patch sysinv 2020-05-19 08:50:54.365 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 08:55:54.370 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-08-55-54 patch sysinv 2020-05-19 08:55:54.371 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:00:54.428 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-00-54 patch sysinv 2020-05-19 09:00:54.429 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:05:54.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-05-54 patch sysinv 2020-05-19 09:05:54.411 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:10:54.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-10-54 patch sysinv 2020-05-19 09:10:54.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:15:54.492 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-15-54 patch sysinv 2020-05-19 09:15:54.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:20:54.529 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-20-54 patch sysinv 2020-05-19 09:20:54.529 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:25:54.630 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-25-54 patch sysinv 2020-05-19 09:25:54.631 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:30:54.750 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-30-54 patch sysinv 2020-05-19 09:30:54.751 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:35:54.722 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-35-54 patch sysinv 2020-05-19 09:35:54.723 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:40:55.030 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-40-55 patch sysinv 2020-05-19 09:40:55.030 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:44:27.495 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 09:45:33.280 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 09:45:33.288 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:45:54.594 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-45-54 patch sysinv 2020-05-19 09:45:54.594 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:46:33.348 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 09:46:33.349 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141262 sec sysinv 2020-05-19 09:46:33.357 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:47:33.418 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 09:47:33.426 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:48:33.485 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 09:48:33.486 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.136537 sec sysinv 2020-05-19 09:48:33.495 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:49:33.552 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 09:49:33.561 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 09:50:33.630 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 09:50:33.631 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144486 sec sysinv 2020-05-19 09:50:33.641 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 09:50:54.694 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-50-54 patch sysinv 2020-05-19 09:50:54.695 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 09:51:08.139 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589881506.95-39302109799788/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589881506.95-39302109799788/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 09:51:08.140 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 09:55:54.795 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-09-55-54 patch sysinv 2020-05-19 09:55:54.795 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:00:54.747 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-00-54 patch sysinv 2020-05-19 10:00:54.747 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:05:54.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-05-54 patch sysinv 2020-05-19 10:05:54.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:10:54.945 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-10-54 patch sysinv 2020-05-19 10:10:54.946 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:15:54.924 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-15-54 patch sysinv 2020-05-19 10:15:54.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:20:54.998 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-20-54 patch sysinv 2020-05-19 10:20:54.998 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:25:54.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-25-54 patch sysinv 2020-05-19 10:25:54.902 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:30:54.967 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-30-54 patch sysinv 2020-05-19 10:30:54.967 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:35:55.037 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-35-55 patch sysinv 2020-05-19 10:35:55.038 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:40:54.931 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-40-54 patch sysinv 2020-05-19 10:40:54.932 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:45:55.068 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-45-55 patch sysinv 2020-05-19 10:45:55.068 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:50:54.988 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-50-54 patch sysinv 2020-05-19 10:50:54.988 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:51:08.170 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 10:52:37.122 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 10:52:37.131 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:53:37.205 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 10:53:37.206 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150313 sec sysinv 2020-05-19 10:53:37.217 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:54:37.285 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 10:54:37.293 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:55:37.356 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 10:55:37.357 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150708 sec sysinv 2020-05-19 10:55:37.365 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:55:55.154 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-10-55-55 patch sysinv 2020-05-19 10:55:55.154 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 10:56:37.427 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 10:56:37.436 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 10:57:37.497 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 10:57:37.498 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141534 sec sysinv 2020-05-19 10:57:37.510 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 10:57:47.779 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589885506.57-129942538373345/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589885506.57-129942538373345/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 10:57:47.781 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 11:00:55.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-00-55 patch sysinv 2020-05-19 11:00:55.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:05:55.374 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-05-55 patch sysinv 2020-05-19 11:05:55.374 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:10:55.166 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-10-55 patch sysinv 2020-05-19 11:10:55.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:15:55.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-15-55 patch sysinv 2020-05-19 11:15:55.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:20:55.260 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-20-55 patch sysinv 2020-05-19 11:20:55.261 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:25:55.254 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-25-55 patch sysinv 2020-05-19 11:25:55.254 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:30:55.324 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-30-55 patch sysinv 2020-05-19 11:30:55.324 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:35:55.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-35-55 patch sysinv 2020-05-19 11:35:55.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:40:55.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-40-55 patch sysinv 2020-05-19 11:40:55.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:45:55.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-45-55 patch sysinv 2020-05-19 11:45:55.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:50:55.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-50-55 patch sysinv 2020-05-19 11:50:55.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:55:55.650 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-11-55-55 patch sysinv 2020-05-19 11:55:55.651 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 11:57:47.785 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 11:59:41.025 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 11:59:41.034 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 12:00:41.104 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 12:00:41.105 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145348 sec sysinv 2020-05-19 12:00:41.116 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 12:00:55.501 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-00-55 patch sysinv 2020-05-19 12:00:55.502 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:01:41.191 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 12:01:41.200 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 12:02:41.261 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 12:02:41.262 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.156691 sec sysinv 2020-05-19 12:02:41.272 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 12:03:41.333 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 12:03:41.344 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 12:04:28.992 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589889507.8-79090897366921/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589889507.8-79090897366921/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 12:04:28.993 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 12:04:31.464 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 50.201753 sec sysinv 2020-05-19 12:05:55.569 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-05-55 patch sysinv 2020-05-19 12:05:55.569 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:10:55.572 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-10-55 patch sysinv 2020-05-19 12:10:55.573 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:15:55.647 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-15-55 patch sysinv 2020-05-19 12:15:55.648 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:20:55.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-20-55 patch sysinv 2020-05-19 12:20:55.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:25:55.681 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-25-55 patch sysinv 2020-05-19 12:25:55.682 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:30:55.694 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-30-55 patch sysinv 2020-05-19 12:30:55.694 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:35:55.745 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-35-55 patch sysinv 2020-05-19 12:35:55.746 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:40:55.740 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-40-55 patch sysinv 2020-05-19 12:40:55.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:45:55.679 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-45-55 patch sysinv 2020-05-19 12:45:55.680 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:50:55.843 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-50-55 patch sysinv 2020-05-19 12:50:55.844 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 12:55:55.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-12-55-55 patch sysinv 2020-05-19 12:55:55.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:00:55.910 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-00-55 patch sysinv 2020-05-19 13:00:55.910 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:04:29.046 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 13:05:34.900 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 13:05:34.910 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:05:55.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-05-55 patch sysinv 2020-05-19 13:05:55.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:06:34.990 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 13:06:34.992 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.162501 sec sysinv 2020-05-19 13:06:35.007 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:07:35.071 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 13:07:35.082 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:08:35.143 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 13:08:35.144 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.151373 sec sysinv 2020-05-19 13:08:35.155 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:09:35.215 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 13:09:35.229 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 13:10:35.291 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 13:10:35.292 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147574 sec sysinv 2020-05-19 13:10:35.302 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 13:10:55.975 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-10-55 patch sysinv 2020-05-19 13:10:55.976 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:11:06.704 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589893505.53-217569142893802/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589893505.53-217569142893802/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 13:11:06.705 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 13:15:55.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-15-55 patch sysinv 2020-05-19 13:15:55.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:20:56.049 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-20-56 patch sysinv 2020-05-19 13:20:56.050 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:25:56.002 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-25-55 patch sysinv 2020-05-19 13:25:56.002 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:30:56.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-30-56 patch sysinv 2020-05-19 13:30:56.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:35:55.997 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-35-55 patch sysinv 2020-05-19 13:35:55.998 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:40:56.013 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-40-55 patch sysinv 2020-05-19 13:40:56.013 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:45:56.421 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-45-56 patch sysinv 2020-05-19 13:45:56.422 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:50:56.145 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-50-56 patch sysinv 2020-05-19 13:50:56.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 13:55:56.251 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-13-55-56 patch sysinv 2020-05-19 13:55:56.252 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:00:56.203 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-00-56 patch sysinv 2020-05-19 14:00:56.204 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:05:56.293 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-05-56 patch sysinv 2020-05-19 14:05:56.293 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:10:56.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-10-56 patch sysinv 2020-05-19 14:10:56.250 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:11:06.733 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 14:12:38.785 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 14:12:38.797 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:13:38.859 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 14:13:38.861 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143411 sec sysinv 2020-05-19 14:13:38.876 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:14:38.937 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 14:14:38.945 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:15:39.008 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 14:15:39.009 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148296 sec sysinv 2020-05-19 14:15:39.017 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:15:56.336 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-15-56 patch sysinv 2020-05-19 14:15:56.337 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:16:39.079 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 14:16:39.098 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 14:17:39.163 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 14:17:39.164 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154182 sec sysinv 2020-05-19 14:17:39.180 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 14:17:46.348 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589897505.21-197985742770788/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589897505.21-197985742770788/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 14:17:46.349 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 14:20:56.355 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-20-56 patch sysinv 2020-05-19 14:20:56.355 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:25:56.431 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-25-56 patch sysinv 2020-05-19 14:25:56.431 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:30:56.403 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-30-56 patch sysinv 2020-05-19 14:30:56.404 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:35:56.519 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-35-56 patch sysinv 2020-05-19 14:35:56.519 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:40:56.459 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-40-56 patch sysinv 2020-05-19 14:40:56.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:45:56.903 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-45-56 patch sysinv 2020-05-19 14:45:56.903 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:50:56.502 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-50-56 patch sysinv 2020-05-19 14:50:56.502 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 14:55:56.549 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-14-55-56 patch sysinv 2020-05-19 14:55:56.549 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:00:56.612 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-00-56 patch sysinv 2020-05-19 15:00:56.613 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:05:56.601 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-05-56 patch sysinv 2020-05-19 15:05:56.602 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:10:56.722 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-10-56 patch sysinv 2020-05-19 15:10:56.723 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:15:56.688 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-15-56 patch sysinv 2020-05-19 15:15:56.689 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:17:46.350 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 15:19:42.717 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 15:19:42.726 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 15:20:42.787 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 15:20:42.788 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.13693 sec sysinv 2020-05-19 15:20:42.797 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 15:20:56.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-20-56 patch sysinv 2020-05-19 15:20:56.774 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:21:42.873 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 15:21:42.886 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 15:22:42.950 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 15:22:42.951 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.162903 sec sysinv 2020-05-19 15:22:42.974 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 15:23:43.036 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 15:23:43.045 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 15:24:23.536 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589901502.41-139545634975249/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589901502.41-139545634975249/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 15:24:23.537 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 15:24:25.983 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 43.030658 sec sysinv 2020-05-19 15:25:56.654 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-25-56 patch sysinv 2020-05-19 15:25:56.655 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:30:56.941 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-30-56 patch sysinv 2020-05-19 15:30:56.942 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:35:56.807 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-35-56 patch sysinv 2020-05-19 15:35:56.808 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:40:56.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-40-56 patch sysinv 2020-05-19 15:40:56.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:45:56.792 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-45-56 patch sysinv 2020-05-19 15:45:56.793 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:50:56.849 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-50-56 patch sysinv 2020-05-19 15:50:56.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 15:55:56.924 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-15-55-56 patch sysinv 2020-05-19 15:55:56.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:00:56.917 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-00-56 patch sysinv 2020-05-19 16:00:56.917 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:05:57.106 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-05-57 patch sysinv 2020-05-19 16:05:57.106 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:10:56.923 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-10-56 patch sysinv 2020-05-19 16:10:56.923 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:15:56.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-15-56 patch sysinv 2020-05-19 16:15:56.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:20:56.979 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-20-56 patch sysinv 2020-05-19 16:20:56.979 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:24:23.591 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 16:25:29.433 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 16:25:29.448 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:25:57.095 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-25-57 patch sysinv 2020-05-19 16:25:57.095 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:26:29.511 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 16:26:29.512 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154511 sec sysinv 2020-05-19 16:26:29.526 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:27:29.588 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 16:27:29.599 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:28:29.662 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 16:28:29.663 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150901 sec sysinv 2020-05-19 16:28:29.676 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:29:29.738 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 16:29:29.749 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 16:30:29.815 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 16:30:29.816 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.152333 sec sysinv 2020-05-19 16:30:29.827 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 16:30:57.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-30-57 patch sysinv 2020-05-19 16:30:57.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:31:02.063 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589905500.93-26891625860211/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589905500.93-26891625860211/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 16:31:02.064 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 16:35:57.139 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-35-57 patch sysinv 2020-05-19 16:35:57.140 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:40:57.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-40-57 patch sysinv 2020-05-19 16:40:57.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:45:57.552 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-45-57 patch sysinv 2020-05-19 16:45:57.552 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:50:57.348 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-50-57 patch sysinv 2020-05-19 16:50:57.349 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 16:55:57.271 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-16-55-57 patch sysinv 2020-05-19 16:55:57.271 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:00:57.268 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-00-57 patch sysinv 2020-05-19 17:00:57.269 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:05:57.300 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-05-57 patch sysinv 2020-05-19 17:05:57.300 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:10:57.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-10-57 patch sysinv 2020-05-19 17:10:57.411 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:15:57.466 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-15-57 patch sysinv 2020-05-19 17:15:57.466 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:20:57.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-20-57 patch sysinv 2020-05-19 17:20:57.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:25:57.423 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-25-57 patch sysinv 2020-05-19 17:25:57.423 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:30:57.525 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-30-57 patch sysinv 2020-05-19 17:30:57.526 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:31:02.091 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 17:32:33.307 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 17:32:33.317 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:33:33.378 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 17:33:33.379 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138829 sec sysinv 2020-05-19 17:33:33.387 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:34:33.448 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 17:34:33.458 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:35:33.524 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 17:35:33.525 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146075 sec sysinv 2020-05-19 17:35:33.534 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:35:57.553 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-35-57 patch sysinv 2020-05-19 17:35:57.554 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:36:33.596 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 17:36:33.607 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 17:37:33.669 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 17:37:33.670 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14471 sec sysinv 2020-05-19 17:37:33.691 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 17:37:39.289 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589909498.17-111607692757556/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589909498.17-111607692757556/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 17:37:39.290 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 17:40:57.602 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-40-57 patch sysinv 2020-05-19 17:40:57.602 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:45:57.947 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-45-57 patch sysinv 2020-05-19 17:45:57.949 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:50:57.574 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-50-57 patch sysinv 2020-05-19 17:50:57.574 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 17:55:57.660 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-17-55-57 patch sysinv 2020-05-19 17:55:57.661 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:00:57.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-00-57 patch sysinv 2020-05-19 18:00:57.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:05:57.722 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-05-57 patch sysinv 2020-05-19 18:05:57.722 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:10:57.719 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-10-57 patch sysinv 2020-05-19 18:10:57.720 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:15:57.802 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-15-57 patch sysinv 2020-05-19 18:15:57.803 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:20:57.761 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-20-57 patch sysinv 2020-05-19 18:20:57.762 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:25:57.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-25-57 patch sysinv 2020-05-19 18:25:57.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:30:57.967 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-30-57 patch sysinv 2020-05-19 18:30:57.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:35:57.868 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-35-57 patch sysinv 2020-05-19 18:35:57.869 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:37:39.291 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 18:39:37.229 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 18:39:37.242 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 18:40:37.305 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 18:40:37.306 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143211 sec sysinv 2020-05-19 18:40:37.314 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 18:40:58.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-40-57 patch sysinv 2020-05-19 18:40:58.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:41:37.374 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 18:41:37.383 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 18:42:37.443 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 18:42:37.444 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138152 sec sysinv 2020-05-19 18:42:37.453 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 18:43:37.513 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 18:43:37.521 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 18:44:16.378 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589913495.22-29506356835375/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589913495.22-29506356835375/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 18:44:16.379 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 18:44:18.658 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 41.213626 sec sysinv 2020-05-19 18:45:57.858 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-45-57 patch sysinv 2020-05-19 18:45:57.859 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:50:58.117 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-50-58 patch sysinv 2020-05-19 18:50:58.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 18:55:57.940 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-18-55-57 patch sysinv 2020-05-19 18:55:57.940 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:00:57.953 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-00-57 patch sysinv 2020-05-19 19:00:57.953 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:05:58.064 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-05-58 patch sysinv 2020-05-19 19:05:58.064 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:10:58.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-10-57 patch sysinv 2020-05-19 19:10:58.020 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:15:58.049 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-15-58 patch sysinv 2020-05-19 19:15:58.050 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:20:58.077 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-20-58 patch sysinv 2020-05-19 19:20:58.077 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:25:58.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-25-58 patch sysinv 2020-05-19 19:25:58.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:30:58.284 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-30-58 patch sysinv 2020-05-19 19:30:58.285 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:35:58.328 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-35-58 patch sysinv 2020-05-19 19:35:58.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:40:58.332 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-40-58 patch sysinv 2020-05-19 19:40:58.332 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:44:16.433 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 19:45:22.098 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 19:45:22.113 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:45:58.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-45-58 patch sysinv 2020-05-19 19:45:58.358 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:46:22.175 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 19:46:22.177 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150147 sec sysinv 2020-05-19 19:46:22.186 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:47:22.243 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 19:47:22.251 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:48:22.312 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 19:48:22.313 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.13581 sec sysinv 2020-05-19 19:48:22.324 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:49:22.383 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 19:49:22.391 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 19:50:22.452 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 19:50:22.454 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.140447 sec sysinv 2020-05-19 19:50:22.468 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 19:50:53.540 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589917492.42-103168720041826/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589917492.42-103168720041826/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 19:50:53.540 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 19:50:58.377 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-50-58 patch sysinv 2020-05-19 19:50:58.378 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 19:55:58.491 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-19-55-58 patch sysinv 2020-05-19 19:55:58.492 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:00:58.360 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-00-58 patch sysinv 2020-05-19 20:00:58.360 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:05:58.407 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-05-58 patch sysinv 2020-05-19 20:05:58.408 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:10:58.422 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-10-58 patch sysinv 2020-05-19 20:10:58.423 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:15:58.575 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-15-58 patch sysinv 2020-05-19 20:15:58.575 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:20:58.557 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-20-58 patch sysinv 2020-05-19 20:20:58.557 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:25:58.534 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-25-58 patch sysinv 2020-05-19 20:25:58.535 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:30:58.642 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-30-58 patch sysinv 2020-05-19 20:30:58.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:35:58.650 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-35-58 patch sysinv 2020-05-19 20:35:58.650 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:40:58.717 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-40-58 patch sysinv 2020-05-19 20:40:58.718 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:45:58.673 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-45-58 patch sysinv 2020-05-19 20:45:58.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:50:53.567 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 20:50:58.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-50-58 patch sysinv 2020-05-19 20:50:58.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:52:25.947 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 20:52:25.977 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:53:26.038 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 20:53:26.039 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.159155 sec sysinv 2020-05-19 20:53:26.050 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:54:26.119 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 20:54:26.136 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:55:26.214 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 20:55:26.215 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.175017 sec sysinv 2020-05-19 20:55:26.223 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:55:58.768 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-20-55-58 patch sysinv 2020-05-19 20:55:58.769 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 20:56:26.284 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 20:56:26.294 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 20:57:26.354 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 20:57:26.355 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139914 sec sysinv 2020-05-19 20:57:26.364 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 20:57:31.808 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589921490.51-32234763814145/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589921490.51-32234763814145/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 20:57:31.809 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 21:00:58.929 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-00-58 patch sysinv 2020-05-19 21:00:58.929 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:05:58.800 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-05-58 patch sysinv 2020-05-19 21:05:58.800 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:10:58.985 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-10-58 patch sysinv 2020-05-19 21:10:58.986 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:15:58.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-15-58 patch sysinv 2020-05-19 21:15:58.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:20:58.894 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-20-58 patch sysinv 2020-05-19 21:20:58.895 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:25:58.986 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-25-58 patch sysinv 2020-05-19 21:25:58.986 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:30:58.937 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-30-58 patch sysinv 2020-05-19 21:30:58.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:35:59.057 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-35-59 patch sysinv 2020-05-19 21:35:59.059 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:40:58.926 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-40-58 patch sysinv 2020-05-19 21:40:58.927 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:45:58.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-45-58 patch sysinv 2020-05-19 21:45:58.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:50:59.110 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-50-59 patch sysinv 2020-05-19 21:50:59.111 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:55:59.024 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-21-55-58 patch sysinv 2020-05-19 21:55:59.025 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 21:57:31.811 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 21:59:29.897 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 21:59:29.907 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 22:00:29.976 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 22:00:29.977 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146853 sec sysinv 2020-05-19 22:00:29.986 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 22:00:59.101 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-00-59 patch sysinv 2020-05-19 22:00:59.101 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:01:30.048 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 22:01:30.057 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 22:02:30.132 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 22:02:30.134 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.155823 sec sysinv 2020-05-19 22:02:30.142 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 22:03:30.202 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 22:03:30.212 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 22:04:10.200 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589925489.09-278406677756951/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589925489.09-278406677756951/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 22:04:10.201 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 22:04:12.606 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 42.472109 sec sysinv 2020-05-19 22:05:59.085 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-05-59 patch sysinv 2020-05-19 22:05:59.086 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:10:59.213 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-10-59 patch sysinv 2020-05-19 22:10:59.213 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:15:59.268 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-15-59 patch sysinv 2020-05-19 22:15:59.269 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:20:59.229 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-20-59 patch sysinv 2020-05-19 22:20:59.229 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:25:59.303 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-25-59 patch sysinv 2020-05-19 22:25:59.304 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:30:59.257 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-30-59 patch sysinv 2020-05-19 22:30:59.258 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:35:59.376 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-35-59 patch sysinv 2020-05-19 22:35:59.377 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:40:59.396 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-40-59 patch sysinv 2020-05-19 22:40:59.396 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:45:59.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-45-59 patch sysinv 2020-05-19 22:45:59.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:50:59.450 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-50-59 patch sysinv 2020-05-19 22:50:59.450 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 22:55:59.475 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-22-55-59 patch sysinv 2020-05-19 22:55:59.476 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:00:59.587 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-00-59 patch sysinv 2020-05-19 23:00:59.587 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:04:10.253 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-19 23:05:16.049 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 23:05:16.057 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:05:59.494 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-05-59 patch sysinv 2020-05-19 23:05:59.494 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:06:16.127 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 23:06:16.129 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.149193 sec sysinv 2020-05-19 23:06:16.147 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:07:16.220 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 23:07:16.230 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:08:16.300 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 23:08:16.302 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.172936 sec sysinv 2020-05-19 23:08:16.312 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:09:16.376 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-19 23:09:16.398 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-19 23:10:16.461 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-19 23:10:16.462 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.159914 sec sysinv 2020-05-19 23:10:16.480 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-19 23:10:47.755 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589929486.67-170042488188760/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589929486.67-170042488188760/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-19 23:10:47.756 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-19 23:10:59.522 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-10-59 patch sysinv 2020-05-19 23:10:59.522 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:15:59.609 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-15-59 patch sysinv 2020-05-19 23:15:59.609 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:20:59.585 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-20-59 patch sysinv 2020-05-19 23:20:59.586 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:25:59.799 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-25-59 patch sysinv 2020-05-19 23:25:59.799 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:30:59.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-30-59 patch sysinv 2020-05-19 23:30:59.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:35:59.692 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-35-59 patch sysinv 2020-05-19 23:35:59.692 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:40:59.781 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-40-59 patch sysinv 2020-05-19 23:40:59.781 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:46:00.098 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-46-00 patch sysinv 2020-05-19 23:46:00.099 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:50:59.701 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-50-59 patch sysinv 2020-05-19 23:50:59.702 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-19 23:55:59.731 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-19-23-55-59 patch sysinv 2020-05-19 23:55:59.732 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:00:59.844 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-00-59 patch sysinv 2020-05-20 00:00:59.845 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:05:59.885 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-05-59 patch sysinv 2020-05-20 00:05:59.886 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:10:47.783 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 00:10:59.903 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-10-59 patch sysinv 2020-05-20 00:10:59.904 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:12:19.955 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 00:12:19.964 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:13:20.030 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 00:13:20.031 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.14611 sec sysinv 2020-05-20 00:13:20.040 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:14:20.105 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 00:14:20.128 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:15:20.194 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 00:15:20.196 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.163728 sec sysinv 2020-05-20 00:15:20.211 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:15:59.993 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-15-59 patch sysinv 2020-05-20 00:15:59.994 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:16:20.271 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 00:16:20.279 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 00:17:20.340 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 00:17:20.341 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145347 sec sysinv 2020-05-20 00:17:20.375 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 00:17:28.122 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589933486.92-223677072458432/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589933486.92-223677072458432/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 00:17:28.123 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 00:21:00.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-20-59 patch sysinv 2020-05-20 00:21:00.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:26:00.035 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-26-00 patch sysinv 2020-05-20 00:26:00.036 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:30:59.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-30-59 patch sysinv 2020-05-20 00:30:59.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:35:59.985 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-35-59 patch sysinv 2020-05-20 00:35:59.985 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:41:00.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-40-59 patch sysinv 2020-05-20 00:41:00.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:46:00.423 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-46-00 patch sysinv 2020-05-20 00:46:00.423 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:51:00.221 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-51-00 patch sysinv 2020-05-20 00:51:00.221 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 00:56:00.163 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-00-56-00 patch sysinv 2020-05-20 00:56:00.163 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:01:00.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-01-00 patch sysinv 2020-05-20 01:01:00.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:06:00.278 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-06-00 patch sysinv 2020-05-20 01:06:00.279 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:11:00.317 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-11-00 patch sysinv 2020-05-20 01:11:00.318 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:16:00.341 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-16-00 patch sysinv 2020-05-20 01:16:00.342 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:17:28.134 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 01:19:23.873 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 01:19:23.882 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 01:20:23.945 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 01:20:23.947 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137225 sec sysinv 2020-05-20 01:20:23.961 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 01:21:00.405 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-21-00 patch sysinv 2020-05-20 01:21:00.405 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:21:24.032 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 01:21:24.048 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 01:22:24.128 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 01:22:24.129 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.182484 sec sysinv 2020-05-20 01:22:24.141 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 01:23:24.203 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 01:23:24.213 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 01:24:06.702 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589937485.54-217350746804113/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589937485.54-217350746804113/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 01:24:06.703 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 01:24:08.852 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 44.722316 sec sysinv 2020-05-20 01:26:00.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-26-00 patch sysinv 2020-05-20 01:26:00.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:31:00.477 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-31-00 patch sysinv 2020-05-20 01:31:00.477 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:36:00.432 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-36-00 patch sysinv 2020-05-20 01:36:00.433 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:41:00.497 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-41-00 patch sysinv 2020-05-20 01:41:00.498 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:46:00.409 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-46-00 patch sysinv 2020-05-20 01:46:00.409 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:51:00.563 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-51-00 patch sysinv 2020-05-20 01:51:00.563 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 01:56:00.569 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-01-56-00 patch sysinv 2020-05-20 01:56:00.570 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:01:00.644 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-01-00 patch sysinv 2020-05-20 02:01:00.644 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:06:00.617 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-06-00 patch sysinv 2020-05-20 02:06:00.617 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:11:00.781 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-11-00 patch sysinv 2020-05-20 02:11:00.782 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:16:00.698 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-16-00 patch sysinv 2020-05-20 02:16:00.699 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:21:00.761 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-21-00 patch sysinv 2020-05-20 02:21:00.761 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:24:06.757 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 02:25:12.294 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 02:25:12.302 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:26:00.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-26-00 patch sysinv 2020-05-20 02:26:00.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:26:12.364 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 02:26:12.366 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138193 sec sysinv 2020-05-20 02:26:12.376 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:27:12.438 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 02:27:12.447 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:28:12.513 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 02:28:12.514 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148418 sec sysinv 2020-05-20 02:28:12.528 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:29:12.595 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 02:29:12.603 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 02:30:12.665 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 02:30:12.667 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.151993 sec sysinv 2020-05-20 02:30:12.687 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 02:30:44.297 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589941483.12-54496446286437/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589941483.12-54496446286437/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 02:30:44.298 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 02:31:00.794 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-31-00 patch sysinv 2020-05-20 02:31:00.794 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:36:00.875 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-36-00 patch sysinv 2020-05-20 02:36:00.876 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:41:00.834 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-41-00 patch sysinv 2020-05-20 02:41:00.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:46:00.815 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-46-00 patch sysinv 2020-05-20 02:46:00.816 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:51:01.074 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-51-00 patch sysinv 2020-05-20 02:51:01.075 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 02:56:00.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-02-56-00 patch sysinv 2020-05-20 02:56:00.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:01:00.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-01-00 patch sysinv 2020-05-20 03:01:00.916 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:06:01.121 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-06-00 patch sysinv 2020-05-20 03:06:01.122 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:11:00.980 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-11-00 patch sysinv 2020-05-20 03:11:00.980 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:16:01.105 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-16-01 patch sysinv 2020-05-20 03:16:01.105 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:21:00.984 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-21-00 patch sysinv 2020-05-20 03:21:00.985 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:26:01.140 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-26-01 patch sysinv 2020-05-20 03:26:01.141 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:30:44.325 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 03:31:01.156 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-31-01 patch sysinv 2020-05-20 03:31:01.157 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:32:16.158 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 03:32:16.174 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:33:16.238 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 03:33:16.239 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.149254 sec sysinv 2020-05-20 03:33:16.250 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:34:16.314 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 03:34:16.323 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:35:16.385 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 03:35:16.387 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.147404 sec sysinv 2020-05-20 03:35:16.408 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:36:01.261 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-36-01 patch sysinv 2020-05-20 03:36:01.262 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:36:16.471 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 03:36:16.480 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 03:37:16.541 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 03:37:16.542 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154383 sec sysinv 2020-05-20 03:37:16.550 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 03:37:28.595 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589945487.46-113402121185316/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589945487.46-113402121185316/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 03:37:28.596 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 03:41:01.164 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-41-01 patch sysinv 2020-05-20 03:41:01.165 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:46:01.608 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-46-01 patch sysinv 2020-05-20 03:46:01.608 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:51:01.232 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-51-01 patch sysinv 2020-05-20 03:51:01.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 03:56:01.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-03-56-01 patch sysinv 2020-05-20 03:56:01.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:01:01.325 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-01-01 patch sysinv 2020-05-20 04:01:01.325 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:06:01.343 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-06-01 patch sysinv 2020-05-20 04:06:01.344 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:11:01.382 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-11-01 patch sysinv 2020-05-20 04:11:01.382 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:16:01.403 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-16-01 patch sysinv 2020-05-20 04:16:01.404 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:21:01.427 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-21-01 patch sysinv 2020-05-20 04:21:01.428 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:26:01.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-26-01 patch sysinv 2020-05-20 04:26:01.384 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:31:01.591 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-31-01 patch sysinv 2020-05-20 04:31:01.592 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:36:01.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-36-01 patch sysinv 2020-05-20 04:36:01.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:37:28.604 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 04:39:20.068 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 04:39:20.082 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 04:40:20.145 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 04:40:20.146 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.145361 sec sysinv 2020-05-20 04:40:20.162 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 04:41:01.646 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-41-01 patch sysinv 2020-05-20 04:41:01.646 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:41:20.224 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 04:41:20.238 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 04:42:20.299 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 04:42:20.301 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.153813 sec sysinv 2020-05-20 04:42:20.312 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 04:43:20.374 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 04:43:20.382 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 04:44:05.854 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589949484.65-253079878871001/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589949484.65-253079878871001/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 04:44:05.855 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 04:44:08.140 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 47.838537 sec sysinv 2020-05-20 04:46:01.498 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-46-01 patch sysinv 2020-05-20 04:46:01.498 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:51:01.703 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-51-01 patch sysinv 2020-05-20 04:51:01.703 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 04:56:01.651 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-04-56-01 patch sysinv 2020-05-20 04:56:01.651 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:01:01.655 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-01-01 patch sysinv 2020-05-20 05:01:01.656 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:06:01.693 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-06-01 patch sysinv 2020-05-20 05:06:01.694 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:11:01.691 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-11-01 patch sysinv 2020-05-20 05:11:01.692 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:16:01.881 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-16-01 patch sysinv 2020-05-20 05:16:01.882 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:21:01.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-21-01 patch sysinv 2020-05-20 05:21:01.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:26:01.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-26-01 patch sysinv 2020-05-20 05:26:01.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:31:01.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-31-01 patch sysinv 2020-05-20 05:31:01.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:36:01.878 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-36-01 patch sysinv 2020-05-20 05:36:01.879 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:41:01.938 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-41-01 patch sysinv 2020-05-20 05:41:01.939 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:44:05.907 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 05:45:11.575 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 05:45:11.592 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:46:01.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-46-01 patch sysinv 2020-05-20 05:46:01.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:46:11.660 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 05:46:11.661 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.153625 sec sysinv 2020-05-20 05:46:11.671 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:47:11.737 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 05:47:11.750 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:48:11.816 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 05:48:11.817 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.15584 sec sysinv 2020-05-20 05:48:11.827 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:49:11.889 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 05:49:11.903 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 05:50:11.964 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 05:50:11.966 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148367 sec sysinv 2020-05-20 05:50:11.989 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 05:50:49.804 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589953488.66-71803996260724/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589953488.66-71803996260724/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 05:50:49.804 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 05:51:02.045 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-51-02 patch sysinv 2020-05-20 05:51:02.046 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 05:56:01.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-05-56-01 patch sysinv 2020-05-20 05:56:01.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:01:01.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-01-01 patch sysinv 2020-05-20 06:01:01.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:06:02.144 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-06-02 patch sysinv 2020-05-20 06:06:02.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:11:02.022 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-11-01 patch sysinv 2020-05-20 06:11:02.023 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:16:02.065 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-16-02 patch sysinv 2020-05-20 06:16:02.066 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:21:02.132 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-21-02 patch sysinv 2020-05-20 06:21:02.133 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:26:02.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-26-02 patch sysinv 2020-05-20 06:26:02.234 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:31:02.216 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-31-02 patch sysinv 2020-05-20 06:31:02.216 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:36:02.309 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-36-02 patch sysinv 2020-05-20 06:36:02.309 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:41:02.263 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-41-02 patch sysinv 2020-05-20 06:41:02.263 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:46:02.385 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-46-02 patch sysinv 2020-05-20 06:46:02.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:50:49.836 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 06:51:02.356 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-51-02 patch sysinv 2020-05-20 06:51:02.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:52:15.447 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 06:52:15.460 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:53:15.522 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 06:53:15.523 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143157 sec sysinv 2020-05-20 06:53:15.533 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:54:15.604 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 06:54:15.613 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:55:15.685 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 06:55:15.686 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.163357 sec sysinv 2020-05-20 06:55:15.705 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:56:02.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-06-56-02 patch sysinv 2020-05-20 06:56:02.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 06:56:15.767 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 06:56:15.779 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 06:57:15.840 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 06:57:15.841 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154866 sec sysinv 2020-05-20 06:57:15.849 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 06:57:26.371 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589957485.22-147040369166622/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589957485.22-147040369166622/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 06:57:26.372 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 07:01:02.375 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-01-02 patch sysinv 2020-05-20 07:01:02.375 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:06:02.398 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-06-02 patch sysinv 2020-05-20 07:06:02.399 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:11:02.494 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-11-02 patch sysinv 2020-05-20 07:11:02.495 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:16:02.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-16-02 patch sysinv 2020-05-20 07:16:02.482 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:21:02.532 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-21-02 patch sysinv 2020-05-20 07:21:02.533 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:26:02.520 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-26-02 patch sysinv 2020-05-20 07:26:02.520 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:31:02.581 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-31-02 patch sysinv 2020-05-20 07:31:02.582 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:36:02.591 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-36-02 patch sysinv 2020-05-20 07:36:02.591 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:41:02.830 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-41-02 patch sysinv 2020-05-20 07:41:02.831 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:46:02.589 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-46-02 patch sysinv 2020-05-20 07:46:02.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:51:02.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-51-02 patch sysinv 2020-05-20 07:51:02.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:56:02.775 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-07-56-02 patch sysinv 2020-05-20 07:56:02.775 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 07:57:26.377 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 07:59:19.373 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 07:59:19.386 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 08:00:19.452 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 08:00:19.454 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.146999 sec sysinv 2020-05-20 08:00:19.467 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 08:01:02.757 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-01-02 patch sysinv 2020-05-20 08:01:02.758 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:01:19.528 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 08:01:19.541 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 08:02:19.604 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 08:02:19.604 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.150537 sec sysinv 2020-05-20 08:02:19.614 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 08:03:19.683 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 08:03:19.705 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 08:04:05.678 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589961484.5-217018837405406/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589961484.5-217018837405406/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 08:04:05.679 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 08:04:07.871 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 48.265837 sec sysinv 2020-05-20 08:06:02.708 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-06-02 patch sysinv 2020-05-20 08:06:02.708 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:11:02.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-11-02 patch sysinv 2020-05-20 08:11:02.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:16:02.840 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-16-02 patch sysinv 2020-05-20 08:16:02.841 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:21:02.890 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-21-02 patch sysinv 2020-05-20 08:21:02.891 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:26:02.887 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-26-02 patch sysinv 2020-05-20 08:26:02.888 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:31:02.987 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-31-02 patch sysinv 2020-05-20 08:31:02.987 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:36:03.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-36-03 patch sysinv 2020-05-20 08:36:03.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:41:03.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-41-03 patch sysinv 2020-05-20 08:41:03.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:46:03.388 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-46-03 patch sysinv 2020-05-20 08:46:03.389 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:51:02.982 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-51-02 patch sysinv 2020-05-20 08:51:02.983 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 08:56:03.133 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-08-56-03 patch sysinv 2020-05-20 08:56:03.133 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:01:03.095 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-01-03 patch sysinv 2020-05-20 09:01:03.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:04:05.733 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 09:05:11.317 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 09:05:11.334 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:06:03.207 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-06-03 patch sysinv 2020-05-20 09:06:03.208 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:06:11.396 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 09:06:11.397 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148647 sec sysinv 2020-05-20 09:06:11.408 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:07:11.493 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 09:07:11.506 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:08:11.567 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 09:08:11.568 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.170511 sec sysinv 2020-05-20 09:08:11.576 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:09:11.636 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 09:09:11.644 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 09:10:11.705 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 09:10:11.706 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137631 sec sysinv 2020-05-20 09:10:11.714 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 09:10:43.054 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589965481.91-108023117686277/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589965481.91-108023117686277/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 09:10:43.055 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 09:11:03.178 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-11-03 patch sysinv 2020-05-20 09:11:03.179 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:16:03.248 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-16-03 patch sysinv 2020-05-20 09:16:03.249 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:21:03.277 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-21-03 patch sysinv 2020-05-20 09:21:03.278 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:26:03.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-26-03 patch sysinv 2020-05-20 09:26:03.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:31:03.322 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-31-03 patch sysinv 2020-05-20 09:31:03.322 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:36:03.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-36-03 patch sysinv 2020-05-20 09:36:03.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:41:03.400 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-41-03 patch sysinv 2020-05-20 09:41:03.400 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:46:03.755 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-46-03 patch sysinv 2020-05-20 09:46:03.756 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:51:03.465 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-51-03 patch sysinv 2020-05-20 09:51:03.466 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 09:56:03.452 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-09-56-03 patch sysinv 2020-05-20 09:56:03.452 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:01:03.548 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-01-03 patch sysinv 2020-05-20 10:01:03.548 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:06:03.504 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-06-03 patch sysinv 2020-05-20 10:06:03.504 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:10:43.082 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 10:11:03.621 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-11-03 patch sysinv 2020-05-20 10:11:03.622 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:12:15.201 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 10:12:15.210 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:13:15.279 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 10:13:15.281 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.148284 sec sysinv 2020-05-20 10:13:15.290 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:14:15.351 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 10:14:15.359 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:15:15.422 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 10:15:15.423 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142107 sec sysinv 2020-05-20 10:15:15.437 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:16:03.579 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-16-03 patch sysinv 2020-05-20 10:16:03.579 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:16:15.497 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 10:16:15.505 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 10:17:15.565 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 10:17:15.566 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.142512 sec sysinv 2020-05-20 10:17:15.574 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 10:17:24.170 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589969482.96-211478603914502/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589969482.96-211478603914502/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 10:17:24.172 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 10:21:03.716 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-21-03 patch sysinv 2020-05-20 10:21:03.717 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:26:03.612 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-26-03 patch sysinv 2020-05-20 10:26:03.612 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:31:03.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-31-03 patch sysinv 2020-05-20 10:31:03.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:36:03.709 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-36-03 patch sysinv 2020-05-20 10:36:03.710 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:41:03.868 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-41-03 patch sysinv 2020-05-20 10:41:03.869 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:46:03.772 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-46-03 patch sysinv 2020-05-20 10:46:03.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:51:03.914 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-51-03 patch sysinv 2020-05-20 10:51:03.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 10:56:03.818 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-10-56-03 patch sysinv 2020-05-20 10:56:03.819 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:01:03.856 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-01-03 patch sysinv 2020-05-20 11:01:03.856 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:06:03.839 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-06-03 patch sysinv 2020-05-20 11:06:03.839 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:11:03.891 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-11-03 patch sysinv 2020-05-20 11:11:03.892 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:16:04.129 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-16-04 patch sysinv 2020-05-20 11:16:04.130 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:17:24.175 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 11:19:19.106 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 11:19:19.123 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 11:20:19.197 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 11:20:19.199 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.159934 sec sysinv 2020-05-20 11:20:19.219 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 11:21:03.907 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-21-03 patch sysinv 2020-05-20 11:21:03.907 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:21:19.282 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 11:21:19.294 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 11:22:19.363 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 11:22:19.364 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.164752 sec sysinv 2020-05-20 11:22:19.374 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 11:23:19.439 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 11:23:19.456 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 11:24:02.884 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589973481.44-165635808818716/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589973481.44-165635808818716/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 11:24:02.886 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 11:24:04.998 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 45.634094 sec sysinv 2020-05-20 11:26:03.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-26-03 patch sysinv 2020-05-20 11:26:03.925 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:31:04.117 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-31-04 patch sysinv 2020-05-20 11:31:04.117 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:36:03.994 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-36-03 patch sysinv 2020-05-20 11:36:03.994 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:41:04.031 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-41-04 patch sysinv 2020-05-20 11:41:04.032 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:46:04.054 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-46-04 patch sysinv 2020-05-20 11:46:04.054 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:51:04.129 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-51-04 patch sysinv 2020-05-20 11:51:04.130 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 11:56:04.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-11-56-04 patch sysinv 2020-05-20 11:56:04.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:01:04.223 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-01-04 patch sysinv 2020-05-20 12:01:04.223 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:06:04.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-06-04 patch sysinv 2020-05-20 12:06:04.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:11:04.272 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-11-04 patch sysinv 2020-05-20 12:11:04.273 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:16:04.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-16-04 patch sysinv 2020-05-20 12:16:04.331 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:21:04.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-21-04 patch sysinv 2020-05-20 12:21:04.340 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:24:02.940 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 12:25:08.446 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 12:25:08.455 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:26:04.369 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-26-04 patch sysinv 2020-05-20 12:26:04.370 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:26:08.515 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 12:26:08.516 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.137045 sec sysinv 2020-05-20 12:26:08.525 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:27:08.586 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 12:27:08.595 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:28:08.657 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 12:28:08.658 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141664 sec sysinv 2020-05-20 12:28:08.667 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:29:08.727 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 12:29:08.735 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 12:30:08.796 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 12:30:08.797 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138349 sec sysinv 2020-05-20 12:30:08.805 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 12:30:41.779 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589977480.57-271693514255828/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589977480.57-271693514255828/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 12:30:41.781 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 12:31:04.413 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-31-04 patch sysinv 2020-05-20 12:31:04.414 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:36:04.431 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-36-04 patch sysinv 2020-05-20 12:36:04.432 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:41:04.454 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-41-04 patch sysinv 2020-05-20 12:41:04.455 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:46:04.867 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-46-04 patch sysinv 2020-05-20 12:46:04.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:51:04.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-51-04 patch sysinv 2020-05-20 12:51:04.518 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 12:56:04.547 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-12-56-04 patch sysinv 2020-05-20 12:56:04.548 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:01:04.615 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-01-04 patch sysinv 2020-05-20 13:01:04.615 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:06:04.577 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-06-04 patch sysinv 2020-05-20 13:06:04.577 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:11:04.704 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-11-04 patch sysinv 2020-05-20 13:11:04.705 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:16:04.685 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-16-04 patch sysinv 2020-05-20 13:16:04.686 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:21:04.646 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-21-04 patch sysinv 2020-05-20 13:21:04.646 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:26:04.707 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-26-04 patch sysinv 2020-05-20 13:26:04.708 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:30:41.809 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 13:31:04.710 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-31-04 patch sysinv 2020-05-20 13:31:04.710 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:32:12.295 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 13:32:12.303 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:33:12.364 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 13:33:12.365 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.136404 sec sysinv 2020-05-20 13:33:12.372 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:34:12.433 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 13:34:12.442 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:35:12.504 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 13:35:12.505 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.139819 sec sysinv 2020-05-20 13:35:12.515 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:36:04.921 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-36-04 patch sysinv 2020-05-20 13:36:04.922 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:36:12.575 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 13:36:12.583 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 13:37:12.643 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 13:37:12.644 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.138776 sec sysinv 2020-05-20 13:37:12.652 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 13:37:20.462 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589981479.32-151903833774322/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589981479.32-151903833774322/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 13:37:20.463 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 13:41:04.895 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-41-04 patch sysinv 2020-05-20 13:41:04.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:46:05.306 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-46-05 patch sysinv 2020-05-20 13:46:05.307 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:51:04.826 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-51-04 patch sysinv 2020-05-20 13:51:04.827 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 13:56:04.844 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-13-56-04 patch sysinv 2020-05-20 13:56:04.845 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:01:04.849 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-01-04 patch sysinv 2020-05-20 14:01:04.849 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:06:04.935 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-06-04 patch sysinv 2020-05-20 14:06:04.936 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:11:05.027 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-11-05 patch sysinv 2020-05-20 14:11:05.027 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:16:04.942 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-16-04 patch sysinv 2020-05-20 14:16:04.943 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:21:04.978 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-21-04 patch sysinv 2020-05-20 14:21:04.978 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:26:04.997 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-26-04 patch sysinv 2020-05-20 14:26:04.997 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:31:05.079 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-31-05 patch sysinv 2020-05-20 14:31:05.080 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:36:05.267 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-36-05 patch sysinv 2020-05-20 14:36:05.268 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:37:20.467 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 14:39:16.177 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 14:39:16.188 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 14:40:16.253 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 14:40:16.255 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.143606 sec sysinv 2020-05-20 14:40:16.275 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 14:41:05.163 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-41-05 patch sysinv 2020-05-20 14:41:05.163 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:41:16.337 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 14:41:16.346 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 14:42:16.409 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 14:42:16.410 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.154672 sec sysinv 2020-05-20 14:42:16.419 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 14:43:16.479 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 14:43:16.488 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 14:44:01.094 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589985479.97-123070875620134/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589985479.97-123070875620134/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 14:44:01.095 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 14:44:03.399 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 46.988159 sec sysinv 2020-05-20 14:46:05.119 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-46-05 patch sysinv 2020-05-20 14:46:05.119 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:51:05.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-51-05 patch sysinv 2020-05-20 14:51:05.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 14:56:05.341 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-14-56-05 patch sysinv 2020-05-20 14:56:05.342 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:01:05.330 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-01-05 patch sysinv 2020-05-20 15:01:05.331 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:06:05.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-06-05 patch sysinv 2020-05-20 15:06:05.358 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:11:05.511 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-11-05 patch sysinv 2020-05-20 15:11:05.512 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:16:05.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-16-05 patch sysinv 2020-05-20 15:16:05.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:21:05.403 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-21-05 patch sysinv 2020-05-20 15:21:05.404 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:26:05.460 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-26-05 patch sysinv 2020-05-20 15:26:05.461 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:31:05.449 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-31-05 patch sysinv 2020-05-20 15:31:05.449 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:32:38.581 91235 WARNING wsme.api [-] Client-side error: Invalid input for field/attribute uuid. Value: 'docker'. unable to convert to uuid. Error: Expected a uuid but received docker.: InvalidInput: Invalid input for field/attribute uuid. Value: 'docker'. unable to convert to uuid. Error: Expected a uuid but received docker. sysinv 2020-05-20 15:36:05.610 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-36-05 patch sysinv 2020-05-20 15:36:05.611 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:41:05.511 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-41-05 patch sysinv 2020-05-20 15:41:05.511 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:44:01.148 90642 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 15:45:06.839 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 15:45:06.848 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:46:05.624 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-46-05 patch sysinv 2020-05-20 15:46:05.624 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:46:06.913 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:46:06.915 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.144702 sec sysinv 2020-05-20 15:46:06.928 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:47:06.990 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 15:47:07.005 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:48:07.069 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:48:07.070 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.155252 sec sysinv 2020-05-20 15:48:07.083 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:49:07.146 10053 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 15:49:07.161 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 15:50:07.236 10053 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:50:07.237 10053 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.165834 sec sysinv 2020-05-20 15:50:07.246 10053 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 15:50:39.120 90642 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1589989477.98-15186953626104/download_images.py\", line 117, in \n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-root/tmp/ansible-tmp-1589989477.98-15186953626104/download_images.py\", line 117, in ", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['quay.io/calico/cni:v3.6.2', 'quay.io/calico/node:v3.6.2', 'quay.io/calico/kube-controllers:v3.6.2', 'docker.io/starlingx/multus:v3.2.16', 'docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest', 'docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest']"], "stdout": " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")\nSleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...\n", "stdout_lines": [" Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/cni:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/cni:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/node:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/node:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: quay.io/calico/kube-controllers:v3.6.2500 Server Error: Internal Server Error (\"Get https://quay.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image quay.io/calico/kube-controllers:v3.6.2 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/multus:v3.2.16500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/multus:v3.2.16 ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ...", " Image download failed: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest500 Server Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-chain.intel.com on 192.168.204.1:53: no such host\")", "Sleep 20s before retry downloading image docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest ..."]} PLAY RECAP ********************************************************************* localhost : ok=37 changed=9 unreachable=0 failed=1 . sysinv 2020-05-20 15:50:39.122 90642 ERROR sysinv.conductor.manager [-] Failed to upgrade/downgrade kubernetes networking images: ansible-playbook returned an error: 2: Exception: ansible-playbook returned an error: 2 sysinv 2020-05-20 15:51:05.579 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-51-05 patch sysinv 2020-05-20 15:51:05.579 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:56:05.690 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-56-05 patch sysinv 2020-05-20 15:56:05.690 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:39.607 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:57:39.608 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:57:39.608 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:57:39.609 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:57:39.609 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:57:41.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:41.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:57:41.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:41.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:41.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-20 15:57:41.256 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:41.307 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:41.307 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:57:41.366 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-57-41 patch sysinv 2020-05-20 15:57:43.918 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-43 patch sysinv 2020-05-20 15:57:43.918 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:43.964 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-43 patch sysinv 2020-05-20 15:57:43.964 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:53.934 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-53 patch sysinv 2020-05-20 15:57:53.934 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:53.982 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-53 patch sysinv 2020-05-20 15:57:53.983 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:56.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-56 patch sysinv 2020-05-20 15:57:56.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:57:56.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-57-56 patch sysinv 2020-05-20 15:57:56.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-56 patch sysinv 2020-05-20 15:57:56.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:57:56.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-57-56 patch sysinv 2020-05-20 15:57:56.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:57:56.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-57-56 patch sysinv 2020-05-20 15:58:03.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-03 patch sysinv 2020-05-20 15:58:03.957 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:04.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-03 patch sysinv 2020-05-20 15:58:04.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:13.977 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-13 patch sysinv 2020-05-20 15:58:13.978 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:14.024 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-14 patch sysinv 2020-05-20 15:58:14.024 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:15.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-15 patch sysinv 2020-05-20 15:58:15.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:15.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-15 patch sysinv 2020-05-20 15:58:15.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-15 patch sysinv 2020-05-20 15:58:15.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:15.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-15 patch sysinv 2020-05-20 15:58:15.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:15.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-15 patch sysinv 2020-05-20 15:58:16.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-16 patch sysinv 2020-05-20 15:58:16.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:16.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-16 patch sysinv 2020-05-20 15:58:16.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-16 patch sysinv 2020-05-20 15:58:16.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:16.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-16 patch sysinv 2020-05-20 15:58:16.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:16.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-16 patch sysinv 2020-05-20 15:58:24.005 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-23 patch sysinv 2020-05-20 15:58:24.005 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:24.045 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-24 patch sysinv 2020-05-20 15:58:24.046 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:30.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-30 patch sysinv 2020-05-20 15:58:30.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:30.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-30 patch sysinv 2020-05-20 15:58:30.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-30 patch sysinv 2020-05-20 15:58:30.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:30.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-30 patch sysinv 2020-05-20 15:58:30.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:30.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-30 patch sysinv 2020-05-20 15:58:34.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-34 patch sysinv 2020-05-20 15:58:34.020 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:34.070 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-34 patch sysinv 2020-05-20 15:58:34.071 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:36.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-36 patch sysinv 2020-05-20 15:58:36.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:36.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-36 patch sysinv 2020-05-20 15:58:36.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-36 patch sysinv 2020-05-20 15:58:36.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:36.293 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-36 patch sysinv 2020-05-20 15:58:36.293 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:36.448 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-36 patch sysinv 2020-05-20 15:58:39.792 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:58:39.793 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:58:39.794 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:58:39.794 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:58:39.795 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:58:44.042 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-44 patch sysinv 2020-05-20 15:58:44.043 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:44.087 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-44 patch sysinv 2020-05-20 15:58:44.087 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:49.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-49 patch sysinv 2020-05-20 15:58:49.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:49.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-49 patch sysinv 2020-05-20 15:58:49.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-49 patch sysinv 2020-05-20 15:58:49.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:49.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-49 patch sysinv 2020-05-20 15:58:49.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:49.363 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-49 patch sysinv 2020-05-20 15:58:54.060 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-54 patch sysinv 2020-05-20 15:58:54.060 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:54.109 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-54 patch sysinv 2020-05-20 15:58:54.110 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:56.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-56 patch sysinv 2020-05-20 15:58:56.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:56.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-56 patch sysinv 2020-05-20 15:58:56.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-56 patch sysinv 2020-05-20 15:58:56.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:58:56.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-58-56 patch sysinv 2020-05-20 15:58:56.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:58:56.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-58-56 patch sysinv 2020-05-20 15:59:04.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-04 patch sysinv 2020-05-20 15:59:04.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:04.133 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-04 patch sysinv 2020-05-20 15:59:04.134 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:05.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-05 patch sysinv 2020-05-20 15:59:05.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:05.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-05 patch sysinv 2020-05-20 15:59:05.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-05 patch sysinv 2020-05-20 15:59:05.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:05.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-05 patch sysinv 2020-05-20 15:59:05.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:05.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-05 patch sysinv 2020-05-20 15:59:14.116 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-14 patch sysinv 2020-05-20 15:59:14.116 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:14.150 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-14 patch sysinv 2020-05-20 15:59:14.150 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:17.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-17 patch sysinv 2020-05-20 15:59:17.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:17.193 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-17 patch sysinv 2020-05-20 15:59:17.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-17 patch sysinv 2020-05-20 15:59:17.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:17.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-17 patch sysinv 2020-05-20 15:59:17.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:17.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-17 patch sysinv 2020-05-20 15:59:24.126 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.127 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:24.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:24.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.216 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:24.233 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.266 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.266 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:24.319 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:24.319 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:24.374 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-24 patch sysinv 2020-05-20 15:59:34.143 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-34 patch sysinv 2020-05-20 15:59:34.143 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:34.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-34 patch sysinv 2020-05-20 15:59:34.195 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:38.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-38 patch sysinv 2020-05-20 15:59:38.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:38.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-38 patch sysinv 2020-05-20 15:59:38.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-38 patch sysinv 2020-05-20 15:59:38.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:38.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-38 patch sysinv 2020-05-20 15:59:38.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:38.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-38 patch sysinv 2020-05-20 15:59:39.662 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:59:39.663 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:59:39.664 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 15:59:39.664 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 15:59:39.665 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 15:59:40.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-40 patch sysinv 2020-05-20 15:59:40.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:40.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-40 patch sysinv 2020-05-20 15:59:40.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-40 patch sysinv 2020-05-20 15:59:40.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:40.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-40 patch sysinv 2020-05-20 15:59:40.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:40.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-40 patch sysinv 2020-05-20 15:59:44.166 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-44 patch sysinv 2020-05-20 15:59:44.166 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:44.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-44 patch sysinv 2020-05-20 15:59:44.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:54.184 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-54 patch sysinv 2020-05-20 15:59:54.185 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:54.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-54 patch sysinv 2020-05-20 15:59:54.233 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:58.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-58 patch sysinv 2020-05-20 15:59:58.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:58.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-58 patch sysinv 2020-05-20 15:59:58.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-58 patch sysinv 2020-05-20 15:59:58.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:58.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-58 patch sysinv 2020-05-20 15:59:58.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:58.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-58 patch sysinv 2020-05-20 15:59:59.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-59 patch sysinv 2020-05-20 15:59:59.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:59.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-59 patch sysinv 2020-05-20 15:59:59.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-59 patch sysinv 2020-05-20 15:59:59.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 15:59:59.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-15-59-59 patch sysinv 2020-05-20 15:59:59.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 15:59:59.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-15-59-59 patch sysinv 2020-05-20 16:00:04.213 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-04 patch sysinv 2020-05-20 16:00:04.214 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:04.259 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-04 patch sysinv 2020-05-20 16:00:04.259 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:14.182 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.183 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:14.201 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.256 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.256 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:14.305 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.305 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:14.363 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.427 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.427 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:14.478 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-14 patch sysinv 2020-05-20 16:00:14.478 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:19.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-19 patch sysinv 2020-05-20 16:00:19.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:19.185 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-19 patch sysinv 2020-05-20 16:00:19.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-19 patch sysinv 2020-05-20 16:00:19.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:19.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-19 patch sysinv 2020-05-20 16:00:19.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:19.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-19 patch sysinv 2020-05-20 16:00:24.246 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-24 patch sysinv 2020-05-20 16:00:24.246 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:24.295 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-24 patch sysinv 2020-05-20 16:00:24.295 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:34.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:34.295 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.335 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.335 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:34.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.386 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:34.442 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.507 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.508 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:34.558 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-34 patch sysinv 2020-05-20 16:00:34.558 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:39.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-39 patch sysinv 2020-05-20 16:00:39.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:39.700 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-39 patch sysinv 2020-05-20 16:00:39.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-39 patch sysinv 2020-05-20 16:00:39.740 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:39.757 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:00:39.758 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:00:39.758 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:00:39.759 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:00:39.760 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:00:39.795 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-39 patch sysinv 2020-05-20 16:00:39.796 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:39.868 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-39 patch sysinv 2020-05-20 16:00:44.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-44 patch sysinv 2020-05-20 16:00:44.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:44.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-44 patch sysinv 2020-05-20 16:00:44.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:49.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-49 patch sysinv 2020-05-20 16:00:49.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:49.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-49 patch sysinv 2020-05-20 16:00:49.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-49 patch sysinv 2020-05-20 16:00:49.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:49.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-49 patch sysinv 2020-05-20 16:00:49.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:00:49.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-00-49 patch sysinv 2020-05-20 16:00:54.310 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-54 patch sysinv 2020-05-20 16:00:54.310 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:00:54.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-00-54 patch sysinv 2020-05-20 16:00:54.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:00.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-00 patch sysinv 2020-05-20 16:01:00.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:00.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-00 patch sysinv 2020-05-20 16:01:00.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-00 patch sysinv 2020-05-20 16:01:00.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:00.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-00 patch sysinv 2020-05-20 16:01:00.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:00.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-00 patch sysinv 2020-05-20 16:01:04.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-04 patch sysinv 2020-05-20 16:01:04.329 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:04.382 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-04 patch sysinv 2020-05-20 16:01:04.383 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:09.726 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-09 patch sysinv 2020-05-20 16:01:09.727 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:09.744 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-09 patch sysinv 2020-05-20 16:01:09.795 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-09 patch sysinv 2020-05-20 16:01:09.796 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:09.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-09 patch sysinv 2020-05-20 16:01:09.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:09.903 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-09 patch sysinv 2020-05-20 16:01:14.365 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-14 patch sysinv 2020-05-20 16:01:14.365 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:14.419 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-14 patch sysinv 2020-05-20 16:01:14.419 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:21.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-21 patch sysinv 2020-05-20 16:01:21.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:21.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-21 patch sysinv 2020-05-20 16:01:21.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-21 patch sysinv 2020-05-20 16:01:21.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:21.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-21 patch sysinv 2020-05-20 16:01:21.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:21.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-21 patch sysinv 2020-05-20 16:01:24.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.372 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:24.422 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.422 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:24.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:24.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:24.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:24.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:24.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-24 patch sysinv 2020-05-20 16:01:34.395 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-34 patch sysinv 2020-05-20 16:01:34.396 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:34.440 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-34 patch sysinv 2020-05-20 16:01:34.440 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:39.727 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:01:39.728 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:01:39.729 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:01:39.729 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:01:39.729 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:01:42.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:42.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:42.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:42.429 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:42.531 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:42.532 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:42.586 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-42 patch sysinv 2020-05-20 16:01:44.413 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-44 patch sysinv 2020-05-20 16:01:44.413 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:44.464 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-44 patch sysinv 2020-05-20 16:01:44.464 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:54.435 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-54 patch sysinv 2020-05-20 16:01:54.435 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:54.481 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-54 patch sysinv 2020-05-20 16:01:54.482 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:59.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-59 patch sysinv 2020-05-20 16:01:59.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:59.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-59 patch sysinv 2020-05-20 16:01:59.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-59 patch sysinv 2020-05-20 16:01:59.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:01:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-01-59 patch sysinv 2020-05-20 16:01:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:01:59.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-01-59 patch sysinv 2020-05-20 16:02:02.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-02 patch sysinv 2020-05-20 16:02:02.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:02.194 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-02 patch sysinv 2020-05-20 16:02:02.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-02 patch sysinv 2020-05-20 16:02:02.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:02.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-02 patch sysinv 2020-05-20 16:02:02.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:02.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-02 patch sysinv 2020-05-20 16:02:04.459 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-04 patch sysinv 2020-05-20 16:02:04.459 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:04.502 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-04 patch sysinv 2020-05-20 16:02:04.503 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:14.489 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-14 patch sysinv 2020-05-20 16:02:14.489 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:14.526 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-14 patch sysinv 2020-05-20 16:02:14.526 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:16.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-16 patch sysinv 2020-05-20 16:02:16.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:16.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-16 patch sysinv 2020-05-20 16:02:16.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-16 patch sysinv 2020-05-20 16:02:16.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:16.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-16 patch sysinv 2020-05-20 16:02:16.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:16.448 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-16 patch sysinv 2020-05-20 16:02:23.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-23 patch sysinv 2020-05-20 16:02:23.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:23.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-23 patch sysinv 2020-05-20 16:02:23.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-23 patch sysinv 2020-05-20 16:02:23.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:23.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-23 patch sysinv 2020-05-20 16:02:23.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:23.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-23 patch sysinv 2020-05-20 16:02:24.496 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-24 patch sysinv 2020-05-20 16:02:24.497 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:24.545 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-24 patch sysinv 2020-05-20 16:02:24.545 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:33.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-33 patch sysinv 2020-05-20 16:02:33.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:33.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-33 patch sysinv 2020-05-20 16:02:33.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-33 patch sysinv 2020-05-20 16:02:33.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:33.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-33 patch sysinv 2020-05-20 16:02:33.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:33.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-33 patch sysinv 2020-05-20 16:02:34.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-34 patch sysinv 2020-05-20 16:02:34.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:34.568 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-34 patch sysinv 2020-05-20 16:02:34.569 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:39.811 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:02:39.812 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:02:39.813 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:02:39.814 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:02:39.814 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:02:43.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-43 patch sysinv 2020-05-20 16:02:43.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:43.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-43 patch sysinv 2020-05-20 16:02:43.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-43 patch sysinv 2020-05-20 16:02:43.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:43.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-43 patch sysinv 2020-05-20 16:02:43.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:43.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-43 patch sysinv 2020-05-20 16:02:44.538 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-44 patch sysinv 2020-05-20 16:02:44.539 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:44.586 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-44 patch sysinv 2020-05-20 16:02:44.586 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:50.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-50 patch sysinv 2020-05-20 16:02:50.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:50.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-50 patch sysinv 2020-05-20 16:02:50.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-50 patch sysinv 2020-05-20 16:02:50.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:50.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-50 patch sysinv 2020-05-20 16:02:50.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:02:50.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-02-50 patch sysinv 2020-05-20 16:02:54.564 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-54 patch sysinv 2020-05-20 16:02:54.565 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:02:54.601 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-02-54 patch sysinv 2020-05-20 16:02:54.601 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:04.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:04.185 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:04.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:04.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.578 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.579 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:04.627 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-04 patch sysinv 2020-05-20 16:03:04.628 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:07.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-07 patch sysinv 2020-05-20 16:03:07.675 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:07.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-07 patch sysinv 2020-05-20 16:03:07.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-07 patch sysinv 2020-05-20 16:03:07.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:07.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-07 patch sysinv 2020-05-20 16:03:07.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:07.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-07 patch sysinv 2020-05-20 16:03:14.619 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-14 patch sysinv 2020-05-20 16:03:14.620 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:14.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-14 patch sysinv 2020-05-20 16:03:14.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:24.616 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.616 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:24.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:24.720 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.721 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:24.744 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:24.838 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:24.838 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:24.901 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-24 patch sysinv 2020-05-20 16:03:25.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-25 patch sysinv 2020-05-20 16:03:25.672 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:25.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-25 patch sysinv 2020-05-20 16:03:25.743 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-25 patch sysinv 2020-05-20 16:03:25.744 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:25.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-25 patch sysinv 2020-05-20 16:03:25.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:25.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-25 patch sysinv 2020-05-20 16:03:34.642 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-34 patch sysinv 2020-05-20 16:03:34.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:34.688 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-34 patch sysinv 2020-05-20 16:03:34.688 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:39.791 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:03:39.792 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:03:39.793 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:03:39.794 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:03:39.794 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:03:42.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-42 patch sysinv 2020-05-20 16:03:42.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:42.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-42 patch sysinv 2020-05-20 16:03:42.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-42 patch sysinv 2020-05-20 16:03:42.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:42.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-42 patch sysinv 2020-05-20 16:03:42.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:42.943 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-42 patch sysinv 2020-05-20 16:03:44.660 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-44 patch sysinv 2020-05-20 16:03:44.660 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:44.705 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-44 patch sysinv 2020-05-20 16:03:44.705 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:45.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-45 patch sysinv 2020-05-20 16:03:45.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:45.687 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-45 patch sysinv 2020-05-20 16:03:45.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-45 patch sysinv 2020-05-20 16:03:45.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:45.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-45 patch sysinv 2020-05-20 16:03:45.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:03:45.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-03-45 patch sysinv 2020-05-20 16:03:54.681 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-54 patch sysinv 2020-05-20 16:03:54.682 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:03:54.729 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-03-54 patch sysinv 2020-05-20 16:03:54.730 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:00.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-00 patch sysinv 2020-05-20 16:04:00.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:00.186 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-00 patch sysinv 2020-05-20 16:04:00.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-00 patch sysinv 2020-05-20 16:04:00.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:00.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-00 patch sysinv 2020-05-20 16:04:00.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:00.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-00 patch sysinv 2020-05-20 16:04:04.699 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-04 patch sysinv 2020-05-20 16:04:04.700 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:04.747 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-04 patch sysinv 2020-05-20 16:04:04.747 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:06.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-06 patch sysinv 2020-05-20 16:04:06.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:06.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-06 patch sysinv 2020-05-20 16:04:06.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-06 patch sysinv 2020-05-20 16:04:06.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:06.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-06 patch sysinv 2020-05-20 16:04:06.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:06.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-06 patch sysinv 2020-05-20 16:04:14.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-14 patch sysinv 2020-05-20 16:04:14.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:14.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-14 patch sysinv 2020-05-20 16:04:14.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:17.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-17 patch sysinv 2020-05-20 16:04:17.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:17.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-17 patch sysinv 2020-05-20 16:04:17.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-17 patch sysinv 2020-05-20 16:04:17.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:17.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-17 patch sysinv 2020-05-20 16:04:17.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:17.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-17 patch sysinv 2020-05-20 16:04:24.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-24 patch sysinv 2020-05-20 16:04:24.742 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:24.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-24 patch sysinv 2020-05-20 16:04:24.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:27.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-27 patch sysinv 2020-05-20 16:04:27.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:27.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-27 patch sysinv 2020-05-20 16:04:27.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-27 patch sysinv 2020-05-20 16:04:27.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:27.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-27 patch sysinv 2020-05-20 16:04:27.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:27.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-27 patch sysinv 2020-05-20 16:04:34.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:34.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:34.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:34.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.765 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.766 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:34.810 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-34 patch sysinv 2020-05-20 16:04:34.810 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:39.877 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:04:39.878 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:04:39.879 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:04:39.879 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:04:39.880 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:04:44.830 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-44 patch sysinv 2020-05-20 16:04:44.831 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:44.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-44 patch sysinv 2020-05-20 16:04:44.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:47.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-47 patch sysinv 2020-05-20 16:04:47.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:47.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-47 patch sysinv 2020-05-20 16:04:47.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-47 patch sysinv 2020-05-20 16:04:47.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:47.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-47 patch sysinv 2020-05-20 16:04:47.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:47.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-47 patch sysinv 2020-05-20 16:04:51.672 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-51 patch sysinv 2020-05-20 16:04:51.673 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:51.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-51 patch sysinv 2020-05-20 16:04:51.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-51 patch sysinv 2020-05-20 16:04:51.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:51.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-51 patch sysinv 2020-05-20 16:04:51.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:04:51.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-04-51 patch sysinv 2020-05-20 16:04:54.808 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-54 patch sysinv 2020-05-20 16:04:54.808 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:04:54.852 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-04-54 patch sysinv 2020-05-20 16:04:54.853 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:04.827 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-04 patch sysinv 2020-05-20 16:05:04.827 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:04.872 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-04 patch sysinv 2020-05-20 16:05:04.872 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:07.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.175 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:07.193 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:07.291 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.291 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:07.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:07.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:07.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:07.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:07.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-07 patch sysinv 2020-05-20 16:05:14.864 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-14 patch sysinv 2020-05-20 16:05:14.864 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:14.913 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-14 patch sysinv 2020-05-20 16:05:14.914 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:24.867 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-24 patch sysinv 2020-05-20 16:05:24.868 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:24.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-24 patch sysinv 2020-05-20 16:05:24.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:25.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-25 patch sysinv 2020-05-20 16:05:25.780 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:25.798 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-25 patch sysinv 2020-05-20 16:05:25.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-25 patch sysinv 2020-05-20 16:05:25.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:25.888 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-25 patch sysinv 2020-05-20 16:05:25.889 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:25.945 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-25 patch sysinv 2020-05-20 16:05:28.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-28 patch sysinv 2020-05-20 16:05:28.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:28.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-28 patch sysinv 2020-05-20 16:05:28.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-28 patch sysinv 2020-05-20 16:05:28.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:28.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-28 patch sysinv 2020-05-20 16:05:28.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:28.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-28 patch sysinv 2020-05-20 16:05:34.888 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-34 patch sysinv 2020-05-20 16:05:34.888 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:34.935 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-34 patch sysinv 2020-05-20 16:05:34.935 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:40.056 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:05:40.057 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:05:40.057 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:05:40.058 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:05:40.058 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:05:42.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-42 patch sysinv 2020-05-20 16:05:42.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:42.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-42 patch sysinv 2020-05-20 16:05:42.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-42 patch sysinv 2020-05-20 16:05:42.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-42 patch sysinv 2020-05-20 16:05:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:42.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-42 patch sysinv 2020-05-20 16:05:44.912 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-44 patch sysinv 2020-05-20 16:05:44.912 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:44.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-44 patch sysinv 2020-05-20 16:05:44.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:48.678 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-48 patch sysinv 2020-05-20 16:05:48.679 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:48.704 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-48 patch sysinv 2020-05-20 16:05:48.759 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-48 patch sysinv 2020-05-20 16:05:48.760 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:48.810 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-48 patch sysinv 2020-05-20 16:05:48.811 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:48.893 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-48 patch sysinv 2020-05-20 16:05:54.087 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-54 patch sysinv 2020-05-20 16:05:54.088 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-20 16:05:54.088 91235 WARNING wsme.api [-] Client-side error: Can not 'Reboot' an 'unlocked' host controller-0; Please 'Lock' first: ClientSideError: Can not 'Reboot' an 'unlocked' host controller-0; Please 'Lock' first sysinv 2020-05-20 16:05:54.932 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-54 patch sysinv 2020-05-20 16:05:54.933 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:54.976 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-54 patch sysinv 2020-05-20 16:05:54.977 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:58.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-58 patch sysinv 2020-05-20 16:05:58.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:58.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-58 patch sysinv 2020-05-20 16:05:58.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-58 patch sysinv 2020-05-20 16:05:58.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:05:58.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-05-58 patch sysinv 2020-05-20 16:05:58.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:05:58.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-05-58 patch sysinv 2020-05-20 16:06:04.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-04 patch sysinv 2020-05-20 16:06:04.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-20 16:06:04.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock sysinv 2020-05-20 16:06:04.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock_controller sysinv 2020-05-20 16:06:04.995 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-04 patch sysinv 2020-05-20 16:06:04.995 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:05.007 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 host check_lock_worker sysinv 2020-05-20 16:06:05.008 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action=lock ihost_val_prenotify: {'ihost_action': 'lock'} ihost_val: {'ihost_action': 'lock', 'task': u'Locking'} sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'lock'} sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=lock, notify_vim=True notify_mtce=True rc=True sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action lock sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_lock_action sysinv 2020-05-20 16:06:05.009 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: lock sysinv 2020-05-20 16:06:05.010 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=True sysinv 2020-05-20 16:06:05.010 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-20 16:06:05.010 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=True sysinv 2020-05-20 16:06:05.010 91235 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'lock'} sysinv 2020-05-20 16:06:05.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'lock', 'task': u'Locking'} sysinv 2020-05-20 16:06:05.020 91235 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=lock sysinv 2020-05-20 16:06:05.020 91235 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=lock sysinv 2020-05-20 16:06:05.020 91235 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=lock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'lock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-20 16:06:05.020 91235 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "lock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-20 16:06:05.029 91235 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-20 16:06:05.045 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-05 patch sysinv 2020-05-20 16:06:05.045 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:05.108 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-05 patch sysinv 2020-05-20 16:06:08.680 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-08 patch sysinv 2020-05-20 16:06:08.681 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:08.709 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-08 patch sysinv 2020-05-20 16:06:08.760 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-08 patch sysinv 2020-05-20 16:06:08.761 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:08.832 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-08 patch sysinv 2020-05-20 16:06:08.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:08.929 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-08 patch sysinv 2020-05-20 16:06:10.071 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-10 patch sysinv 2020-05-20 16:06:10.072 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-20 16:06:10.072 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disable-failed sysinv 2020-05-20 16:06:10.073 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disable-failed ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-20 16:06:10.073 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disable-failed, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-20 16:06:10.073 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:06:10.073 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disable-failed sysinv 2020-05-20 16:06:10.074 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 handle_vim_services_disable_failed ihost_action=lock sysinv 2020-05-20 16:06:10.074 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-disable-failed sysinv 2020-05-20 16:06:10.074 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=True sysinv 2020-05-20 16:06:10.074 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-20 16:06:10.075 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=True sysinv 2020-05-20 16:06:10.075 91235 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': '', 'vim_progress_status': '', 'task': ''} sysinv 2020-05-20 16:06:10.084 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': '', 'vim_progress_status': 'services-disable-failed', 'task': ''} sysinv 2020-05-20 16:06:10.141 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-10 patch sysinv 2020-05-20 16:06:14.971 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-14 patch sysinv 2020-05-20 16:06:14.971 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:15.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:15.019 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:15.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:15.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:15.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:15.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:15.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:15.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:15.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:15.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-15 patch sysinv 2020-05-20 16:06:24.992 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-24 patch sysinv 2020-05-20 16:06:24.992 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:25.039 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-25 patch sysinv 2020-05-20 16:06:25.040 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:28.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-28 patch sysinv 2020-05-20 16:06:28.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:28.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-28 patch sysinv 2020-05-20 16:06:28.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-28 patch sysinv 2020-05-20 16:06:28.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:28.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-28 patch sysinv 2020-05-20 16:06:28.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:28.856 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-28 patch sysinv 2020-05-20 16:06:31.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-31 patch sysinv 2020-05-20 16:06:31.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:31.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-31 patch sysinv 2020-05-20 16:06:31.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-31 patch sysinv 2020-05-20 16:06:31.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:31.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-31 patch sysinv 2020-05-20 16:06:31.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:31.352 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-31 patch sysinv 2020-05-20 16:06:35.014 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-34 patch sysinv 2020-05-20 16:06:35.014 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:35.061 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-35 patch sysinv 2020-05-20 16:06:35.061 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:39.928 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:06:39.929 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:06:39.929 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:06:39.930 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:06:39.930 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:06:45.030 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-45 patch sysinv 2020-05-20 16:06:45.030 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:45.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-45 patch sysinv 2020-05-20 16:06:45.081 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:49.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-49 patch sysinv 2020-05-20 16:06:49.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:49.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-49 patch sysinv 2020-05-20 16:06:49.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-49 patch sysinv 2020-05-20 16:06:49.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:49.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-49 patch sysinv 2020-05-20 16:06:49.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:49.944 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-49 patch sysinv 2020-05-20 16:06:50.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-50 patch sysinv 2020-05-20 16:06:50.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:50.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-50 patch sysinv 2020-05-20 16:06:50.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-50 patch sysinv 2020-05-20 16:06:50.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:50.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-50 patch sysinv 2020-05-20 16:06:50.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:06:50.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-06-50 patch sysinv 2020-05-20 16:06:55.054 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-55 patch sysinv 2020-05-20 16:06:55.054 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:06:55.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-06-55 patch sysinv 2020-05-20 16:06:55.101 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:05.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-05 patch sysinv 2020-05-20 16:07:05.101 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:05.143 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-05 patch sysinv 2020-05-20 16:07:05.143 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:07.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-07 patch sysinv 2020-05-20 16:07:07.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:07.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-07 patch sysinv 2020-05-20 16:07:07.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-07 patch sysinv 2020-05-20 16:07:07.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:07.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-07 patch sysinv 2020-05-20 16:07:07.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:07.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-07 patch sysinv 2020-05-20 16:07:10.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-10 patch sysinv 2020-05-20 16:07:10.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:10.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-10 patch sysinv 2020-05-20 16:07:10.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-10 patch sysinv 2020-05-20 16:07:10.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:10.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-10 patch sysinv 2020-05-20 16:07:10.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:10.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-10 patch sysinv 2020-05-20 16:07:15.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-15 patch sysinv 2020-05-20 16:07:15.096 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:15.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-15 patch sysinv 2020-05-20 16:07:15.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:25.118 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-25 patch sysinv 2020-05-20 16:07:25.119 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:25.165 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-25 patch sysinv 2020-05-20 16:07:25.165 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:26.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-26 patch sysinv 2020-05-20 16:07:26.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:26.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-26 patch sysinv 2020-05-20 16:07:26.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-26 patch sysinv 2020-05-20 16:07:26.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:26.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-26 patch sysinv 2020-05-20 16:07:26.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:26.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-26 patch sysinv 2020-05-20 16:07:31.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-31 patch sysinv 2020-05-20 16:07:31.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:31.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-31 patch sysinv 2020-05-20 16:07:31.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-31 patch sysinv 2020-05-20 16:07:31.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:31.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-31 patch sysinv 2020-05-20 16:07:31.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:31.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-31 patch sysinv 2020-05-20 16:07:35.137 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-35 patch sysinv 2020-05-20 16:07:35.138 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:35.188 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-35 patch sysinv 2020-05-20 16:07:35.188 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:39.910 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:07:39.911 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:07:39.911 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:07:39.912 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:07:39.912 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:07:41.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-41 patch sysinv 2020-05-20 16:07:41.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:41.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-41 patch sysinv 2020-05-20 16:07:41.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-41 patch sysinv 2020-05-20 16:07:41.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:41.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-41 patch sysinv 2020-05-20 16:07:41.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:41.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-41 patch sysinv 2020-05-20 16:07:45.159 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-45 patch sysinv 2020-05-20 16:07:45.160 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:45.207 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-45 patch sysinv 2020-05-20 16:07:45.207 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:52.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-52 patch sysinv 2020-05-20 16:07:52.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:52.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-52 patch sysinv 2020-05-20 16:07:52.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-52 patch sysinv 2020-05-20 16:07:52.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:52.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-52 patch sysinv 2020-05-20 16:07:52.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:07:52.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-07-52 patch sysinv 2020-05-20 16:07:55.180 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-55 patch sysinv 2020-05-20 16:07:55.180 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:07:55.230 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-07-55 patch sysinv 2020-05-20 16:07:55.231 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:00.831 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-00 patch sysinv 2020-05-20 16:08:00.831 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:00.851 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-00 patch sysinv 2020-05-20 16:08:00.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-00 patch sysinv 2020-05-20 16:08:00.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:00.948 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-00 patch sysinv 2020-05-20 16:08:00.948 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:01.003 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-01 patch sysinv 2020-05-20 16:08:05.211 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-05 patch sysinv 2020-05-20 16:08:05.211 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:05.247 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-05 patch sysinv 2020-05-20 16:08:05.247 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:13.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-13 patch sysinv 2020-05-20 16:08:13.672 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:13.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-13 patch sysinv 2020-05-20 16:08:13.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-13 patch sysinv 2020-05-20 16:08:13.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:13.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-13 patch sysinv 2020-05-20 16:08:13.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:13.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-13 patch sysinv 2020-05-20 16:08:15.223 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.224 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:15.268 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.269 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:15.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:15.686 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.840 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.841 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:15.887 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:15.887 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:15.940 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-15 patch sysinv 2020-05-20 16:08:25.250 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-25 patch sysinv 2020-05-20 16:08:25.250 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:25.299 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-25 patch sysinv 2020-05-20 16:08:25.299 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:34.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-34 patch sysinv 2020-05-20 16:08:34.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:34.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-34 patch sysinv 2020-05-20 16:08:34.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-34 patch sysinv 2020-05-20 16:08:34.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:34.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-34 patch sysinv 2020-05-20 16:08:34.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:34.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-34 patch sysinv 2020-05-20 16:08:35.263 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-35 patch sysinv 2020-05-20 16:08:35.263 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:35.317 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-35 patch sysinv 2020-05-20 16:08:35.317 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:36.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-36 patch sysinv 2020-05-20 16:08:36.672 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:36.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-36 patch sysinv 2020-05-20 16:08:36.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-36 patch sysinv 2020-05-20 16:08:36.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:36.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-36 patch sysinv 2020-05-20 16:08:36.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:36.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-36 patch sysinv 2020-05-20 16:08:39.998 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:08:39.999 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:08:40.000 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:08:40.001 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:08:40.001 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:08:45.285 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-45 patch sysinv 2020-05-20 16:08:45.285 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:45.332 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-45 patch sysinv 2020-05-20 16:08:45.333 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:50.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-50 patch sysinv 2020-05-20 16:08:50.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:50.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-50 patch sysinv 2020-05-20 16:08:50.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-50 patch sysinv 2020-05-20 16:08:50.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:50.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-50 patch sysinv 2020-05-20 16:08:50.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:50.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-50 patch sysinv 2020-05-20 16:08:55.306 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.306 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:55.356 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:55.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:55.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:08:55.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:08:55.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:08:55.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-08-55 patch sysinv 2020-05-20 16:09:05.340 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-05 patch sysinv 2020-05-20 16:09:05.341 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:05.374 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-05 patch sysinv 2020-05-20 16:09:05.375 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:11.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-11 patch sysinv 2020-05-20 16:09:11.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:11.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-11 patch sysinv 2020-05-20 16:09:11.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-11 patch sysinv 2020-05-20 16:09:11.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:11.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-11 patch sysinv 2020-05-20 16:09:11.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:11.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-11 patch sysinv 2020-05-20 16:09:15.348 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.349 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:15.394 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.394 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:15.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:15.687 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:15.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:15.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:15.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-15 patch sysinv 2020-05-20 16:09:24.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-24 patch sysinv 2020-05-20 16:09:24.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:24.695 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-24 patch sysinv 2020-05-20 16:09:24.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-24 patch sysinv 2020-05-20 16:09:24.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:24.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-24 patch sysinv 2020-05-20 16:09:24.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:24.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-24 patch sysinv 2020-05-20 16:09:25.367 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-25 patch sysinv 2020-05-20 16:09:25.368 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:25.418 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-25 patch sysinv 2020-05-20 16:09:25.419 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:35.389 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-35 patch sysinv 2020-05-20 16:09:35.390 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:35.438 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-35 patch sysinv 2020-05-20 16:09:35.438 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:36.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-36 patch sysinv 2020-05-20 16:09:36.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:36.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-36 patch sysinv 2020-05-20 16:09:36.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-36 patch sysinv 2020-05-20 16:09:36.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:36.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-36 patch sysinv 2020-05-20 16:09:36.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:36.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-36 patch sysinv 2020-05-20 16:09:39.974 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:09:39.975 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:09:39.976 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:09:39.976 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:09:39.977 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:09:45.486 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.486 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:45.541 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.541 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:45.663 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.664 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:45.684 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:45.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:45.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:45.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-45 patch sysinv 2020-05-20 16:09:55.431 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-55 patch sysinv 2020-05-20 16:09:55.431 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:55.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-55 patch sysinv 2020-05-20 16:09:55.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:56.792 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-56 patch sysinv 2020-05-20 16:09:56.792 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:56.810 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-56 patch sysinv 2020-05-20 16:09:56.856 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-56 patch sysinv 2020-05-20 16:09:56.857 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:56.908 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-56 patch sysinv 2020-05-20 16:09:56.908 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:56.962 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-56 patch sysinv 2020-05-20 16:09:59.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-59 patch sysinv 2020-05-20 16:09:59.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:59.194 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-59 patch sysinv 2020-05-20 16:09:59.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-59 patch sysinv 2020-05-20 16:09:59.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:09:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-09-59 patch sysinv 2020-05-20 16:09:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:09:59.357 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-09-59 patch sysinv 2020-05-20 16:10:05.468 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-05 patch sysinv 2020-05-20 16:10:05.468 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:05.521 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-05 patch sysinv 2020-05-20 16:10:05.521 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:15.475 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-15 patch sysinv 2020-05-20 16:10:15.476 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:15.519 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-15 patch sysinv 2020-05-20 16:10:15.520 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:17.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-17 patch sysinv 2020-05-20 16:10:17.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:17.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-17 patch sysinv 2020-05-20 16:10:17.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-17 patch sysinv 2020-05-20 16:10:17.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:17.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-17 patch sysinv 2020-05-20 16:10:17.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:17.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-17 patch sysinv 2020-05-20 16:10:20.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-20 patch sysinv 2020-05-20 16:10:20.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:20.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-20 patch sysinv 2020-05-20 16:10:20.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-20 patch sysinv 2020-05-20 16:10:20.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:20.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-20 patch sysinv 2020-05-20 16:10:20.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:20.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-20 patch sysinv 2020-05-20 16:10:25.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-25 patch sysinv 2020-05-20 16:10:25.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:25.544 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-25 patch sysinv 2020-05-20 16:10:25.544 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:34.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-34 patch sysinv 2020-05-20 16:10:34.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:34.686 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-34 patch sysinv 2020-05-20 16:10:34.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-34 patch sysinv 2020-05-20 16:10:34.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:34.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-34 patch sysinv 2020-05-20 16:10:34.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:34.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-34 patch sysinv 2020-05-20 16:10:35.512 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-35 patch sysinv 2020-05-20 16:10:35.513 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:35.562 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-35 patch sysinv 2020-05-20 16:10:35.562 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:38.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-38 patch sysinv 2020-05-20 16:10:38.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:38.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-38 patch sysinv 2020-05-20 16:10:38.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-38 patch sysinv 2020-05-20 16:10:38.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:38.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-38 patch sysinv 2020-05-20 16:10:38.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:38.356 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-38 patch sysinv 2020-05-20 16:10:40.060 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:10:40.060 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:10:40.061 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:10:40.062 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:10:40.062 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:10:45.535 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-45 patch sysinv 2020-05-20 16:10:45.535 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:45.586 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-45 patch sysinv 2020-05-20 16:10:45.587 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:55.556 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.557 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:55.604 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.605 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:55.664 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.665 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:55.686 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:55.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:55.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:55.840 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-55 patch sysinv 2020-05-20 16:10:59.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-59 patch sysinv 2020-05-20 16:10:59.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:59.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-59 patch sysinv 2020-05-20 16:10:59.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-59 patch sysinv 2020-05-20 16:10:59.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:10:59.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-10-59 patch sysinv 2020-05-20 16:10:59.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:10:59.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-10-59 patch sysinv 2020-05-20 16:11:05.595 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-05 patch sysinv 2020-05-20 16:11:05.595 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:05.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-05 patch sysinv 2020-05-20 16:11:05.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:08.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-08 patch sysinv 2020-05-20 16:11:08.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:08.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-08 patch sysinv 2020-05-20 16:11:08.750 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-08 patch sysinv 2020-05-20 16:11:08.751 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:08.794 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-08 patch sysinv 2020-05-20 16:11:08.794 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:08.868 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-08 patch sysinv 2020-05-20 16:11:15.599 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-15 patch sysinv 2020-05-20 16:11:15.600 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:15.647 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-15 patch sysinv 2020-05-20 16:11:15.647 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:20.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-20 patch sysinv 2020-05-20 16:11:20.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:20.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-20 patch sysinv 2020-05-20 16:11:20.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-20 patch sysinv 2020-05-20 16:11:20.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:20.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-20 patch sysinv 2020-05-20 16:11:20.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:20.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-20 patch sysinv 2020-05-20 16:11:25.623 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-25 patch sysinv 2020-05-20 16:11:25.623 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:25.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-25 patch sysinv 2020-05-20 16:11:25.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:30.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-30 patch sysinv 2020-05-20 16:11:30.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:30.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-30 patch sysinv 2020-05-20 16:11:30.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-30 patch sysinv 2020-05-20 16:11:30.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:30.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-30 patch sysinv 2020-05-20 16:11:30.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:30.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-30 patch sysinv 2020-05-20 16:11:35.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-35 patch sysinv 2020-05-20 16:11:35.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:35.690 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-35 patch sysinv 2020-05-20 16:11:35.691 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:40.040 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:11:40.041 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:11:40.041 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:11:40.042 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:11:40.042 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:11:40.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-40 patch sysinv 2020-05-20 16:11:40.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:40.687 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-40 patch sysinv 2020-05-20 16:11:40.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-40 patch sysinv 2020-05-20 16:11:40.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:40.785 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-40 patch sysinv 2020-05-20 16:11:40.785 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:40.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-40 patch sysinv 2020-05-20 16:11:42.676 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-42 patch sysinv 2020-05-20 16:11:42.677 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:42.706 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-42 patch sysinv 2020-05-20 16:11:42.758 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-42 patch sysinv 2020-05-20 16:11:42.759 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:42.807 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-42 patch sysinv 2020-05-20 16:11:42.807 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:11:42.869 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-11-42 patch sysinv 2020-05-20 16:11:45.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-45 patch sysinv 2020-05-20 16:11:45.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:45.712 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-45 patch sysinv 2020-05-20 16:11:45.712 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:55.684 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-55 patch sysinv 2020-05-20 16:11:55.685 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:11:55.730 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-11-55 patch sysinv 2020-05-20 16:11:55.730 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:00.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-00 patch sysinv 2020-05-20 16:12:00.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:00.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-00 patch sysinv 2020-05-20 16:12:00.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-00 patch sysinv 2020-05-20 16:12:00.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:00.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-00 patch sysinv 2020-05-20 16:12:00.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:00.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-00 patch sysinv 2020-05-20 16:12:05.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:05.194 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:05.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:05.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.720 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.720 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:05.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-05 patch sysinv 2020-05-20 16:12:05.772 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:15.732 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-15 patch sysinv 2020-05-20 16:12:15.733 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:15.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-15 patch sysinv 2020-05-20 16:12:15.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:17.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-17 patch sysinv 2020-05-20 16:12:17.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:17.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-17 patch sysinv 2020-05-20 16:12:17.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-17 patch sysinv 2020-05-20 16:12:17.242 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:17.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-17 patch sysinv 2020-05-20 16:12:17.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:17.364 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-17 patch sysinv 2020-05-20 16:12:20.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-20 patch sysinv 2020-05-20 16:12:20.837 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:20.855 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-20 patch sysinv 2020-05-20 16:12:20.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-20 patch sysinv 2020-05-20 16:12:20.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:20.950 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-20 patch sysinv 2020-05-20 16:12:20.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:21.006 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-21 patch sysinv 2020-05-20 16:12:25.748 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-25 patch sysinv 2020-05-20 16:12:25.749 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:25.793 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-25 patch sysinv 2020-05-20 16:12:25.793 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:35.766 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-35 patch sysinv 2020-05-20 16:12:35.767 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:35.813 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-35 patch sysinv 2020-05-20 16:12:35.813 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:38.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-38 patch sysinv 2020-05-20 16:12:38.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:38.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-38 patch sysinv 2020-05-20 16:12:38.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-38 patch sysinv 2020-05-20 16:12:38.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:38.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-38 patch sysinv 2020-05-20 16:12:38.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:38.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-38 patch sysinv 2020-05-20 16:12:40.109 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:12:40.110 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:12:40.110 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:12:40.111 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:12:40.111 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:12:40.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-40 patch sysinv 2020-05-20 16:12:40.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:40.854 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-40 patch sysinv 2020-05-20 16:12:40.899 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-40 patch sysinv 2020-05-20 16:12:40.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:40.950 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-40 patch sysinv 2020-05-20 16:12:40.950 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:41.008 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-41 patch sysinv 2020-05-20 16:12:45.791 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-45 patch sysinv 2020-05-20 16:12:45.791 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:45.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-45 patch sysinv 2020-05-20 16:12:45.836 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:51.661 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-51 patch sysinv 2020-05-20 16:12:51.662 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:51.680 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-51 patch sysinv 2020-05-20 16:12:51.732 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-51 patch sysinv 2020-05-20 16:12:51.732 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:51.780 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-51 patch sysinv 2020-05-20 16:12:51.781 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:12:51.836 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-12-51 patch sysinv 2020-05-20 16:12:55.808 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-55 patch sysinv 2020-05-20 16:12:55.808 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:12:55.858 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-12-55 patch sysinv 2020-05-20 16:12:55.858 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:00.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-00 patch sysinv 2020-05-20 16:13:00.834 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:00.853 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-00 patch sysinv 2020-05-20 16:13:00.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-00 patch sysinv 2020-05-20 16:13:00.902 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:00.952 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-00 patch sysinv 2020-05-20 16:13:00.953 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:01.010 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-01 patch sysinv 2020-05-20 16:13:05.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-05 patch sysinv 2020-05-20 16:13:05.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:05.996 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-05 patch sysinv 2020-05-20 16:13:05.997 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:12.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-12 patch sysinv 2020-05-20 16:13:12.673 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:12.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-12 patch sysinv 2020-05-20 16:13:12.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-12 patch sysinv 2020-05-20 16:13:12.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:12.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-12 patch sysinv 2020-05-20 16:13:12.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:12.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-12 patch sysinv 2020-05-20 16:13:15.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-15 patch sysinv 2020-05-20 16:13:15.848 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:15.896 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-15 patch sysinv 2020-05-20 16:13:15.896 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:20.834 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-20 patch sysinv 2020-05-20 16:13:20.834 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:20.856 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-20 patch sysinv 2020-05-20 16:13:20.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-20 patch sysinv 2020-05-20 16:13:20.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:20.952 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-20 patch sysinv 2020-05-20 16:13:20.952 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:21.007 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-21 patch sysinv 2020-05-20 16:13:25.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:25.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:25.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:25.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.871 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:25.922 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-25 patch sysinv 2020-05-20 16:13:25.922 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:35.892 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-35 patch sysinv 2020-05-20 16:13:35.892 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:35.939 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-35 patch sysinv 2020-05-20 16:13:35.940 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:40.088 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:13:40.089 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:13:40.089 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:13:40.090 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:13:40.090 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:13:40.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-40 patch sysinv 2020-05-20 16:13:40.835 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:40.853 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-40 patch sysinv 2020-05-20 16:13:40.900 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-40 patch sysinv 2020-05-20 16:13:40.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:40.951 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-40 patch sysinv 2020-05-20 16:13:40.952 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:41.009 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-41 patch sysinv 2020-05-20 16:13:45.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-45 patch sysinv 2020-05-20 16:13:45.915 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:45.960 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-45 patch sysinv 2020-05-20 16:13:45.960 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:47.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-47 patch sysinv 2020-05-20 16:13:47.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:47.196 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-47 patch sysinv 2020-05-20 16:13:47.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-47 patch sysinv 2020-05-20 16:13:47.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:47.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-47 patch sysinv 2020-05-20 16:13:47.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:47.350 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-47 patch sysinv 2020-05-20 16:13:55.932 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-55 patch sysinv 2020-05-20 16:13:55.933 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:55.983 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-55 patch sysinv 2020-05-20 16:13:55.984 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:59.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-59 patch sysinv 2020-05-20 16:13:59.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:59.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-59 patch sysinv 2020-05-20 16:13:59.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-59 patch sysinv 2020-05-20 16:13:59.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:13:59.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-13-59 patch sysinv 2020-05-20 16:13:59.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:13:59.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-13-59 patch sysinv 2020-05-20 16:14:01.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-01 patch sysinv 2020-05-20 16:14:01.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:01.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-01 patch sysinv 2020-05-20 16:14:01.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-01 patch sysinv 2020-05-20 16:14:01.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:01.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-01 patch sysinv 2020-05-20 16:14:01.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:01.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-01 patch sysinv 2020-05-20 16:14:05.968 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-05 patch sysinv 2020-05-20 16:14:05.969 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:06.022 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-06 patch sysinv 2020-05-20 16:14:06.022 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:15.972 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-15 patch sysinv 2020-05-20 16:14:15.973 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:16.025 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-16 patch sysinv 2020-05-20 16:14:16.025 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:21.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-21 patch sysinv 2020-05-20 16:14:21.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:21.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-21 patch sysinv 2020-05-20 16:14:21.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-21 patch sysinv 2020-05-20 16:14:21.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:21.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-21 patch sysinv 2020-05-20 16:14:21.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:21.847 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-21 patch sysinv 2020-05-20 16:14:22.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-22 patch sysinv 2020-05-20 16:14:22.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:22.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-22 patch sysinv 2020-05-20 16:14:22.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-22 patch sysinv 2020-05-20 16:14:22.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:22.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-22 patch sysinv 2020-05-20 16:14:22.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:22.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-22 patch sysinv 2020-05-20 16:14:25.999 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-25 patch sysinv 2020-05-20 16:14:25.999 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:26.042 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-26 patch sysinv 2020-05-20 16:14:26.042 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:32.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-32 patch sysinv 2020-05-20 16:14:32.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:32.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-32 patch sysinv 2020-05-20 16:14:32.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-32 patch sysinv 2020-05-20 16:14:32.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:32.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-32 patch sysinv 2020-05-20 16:14:32.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:32.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-32 patch sysinv 2020-05-20 16:14:36.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-36 patch sysinv 2020-05-20 16:14:36.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:36.066 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-36 patch sysinv 2020-05-20 16:14:36.067 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:40.174 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:14:40.174 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:14:40.175 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:14:40.176 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:14:40.176 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:14:43.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-43 patch sysinv 2020-05-20 16:14:43.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:43.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-43 patch sysinv 2020-05-20 16:14:43.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-43 patch sysinv 2020-05-20 16:14:43.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:43.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-43 patch sysinv 2020-05-20 16:14:43.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:43.445 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-43 patch sysinv 2020-05-20 16:14:46.131 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-46 patch sysinv 2020-05-20 16:14:46.131 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:46.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-46 patch sysinv 2020-05-20 16:14:46.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:54.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-54 patch sysinv 2020-05-20 16:14:54.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:54.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-54 patch sysinv 2020-05-20 16:14:54.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-54 patch sysinv 2020-05-20 16:14:54.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:54.792 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-54 patch sysinv 2020-05-20 16:14:54.792 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:14:54.863 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-14-54 patch sysinv 2020-05-20 16:14:56.058 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-56 patch sysinv 2020-05-20 16:14:56.059 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:14:56.106 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-14-56 patch sysinv 2020-05-20 16:14:56.106 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:04.176 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-04 patch sysinv 2020-05-20 16:15:04.177 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:04.195 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-04 patch sysinv 2020-05-20 16:15:04.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-04 patch sysinv 2020-05-20 16:15:04.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:04.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-04 patch sysinv 2020-05-20 16:15:04.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:04.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-04 patch sysinv 2020-05-20 16:15:06.104 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-06 patch sysinv 2020-05-20 16:15:06.105 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:06.147 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-06 patch sysinv 2020-05-20 16:15:06.148 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:07.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-07 patch sysinv 2020-05-20 16:15:07.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:07.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-07 patch sysinv 2020-05-20 16:15:07.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-07 patch sysinv 2020-05-20 16:15:07.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:07.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-07 patch sysinv 2020-05-20 16:15:07.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:07.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-07 patch sysinv 2020-05-20 16:15:16.111 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-16 patch sysinv 2020-05-20 16:15:16.112 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:16.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-16 patch sysinv 2020-05-20 16:15:16.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:24.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-24 patch sysinv 2020-05-20 16:15:24.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:24.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-24 patch sysinv 2020-05-20 16:15:24.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-24 patch sysinv 2020-05-20 16:15:24.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:24.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-24 patch sysinv 2020-05-20 16:15:24.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:24.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-24 patch sysinv 2020-05-20 16:15:26.121 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-26 patch sysinv 2020-05-20 16:15:26.121 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:26.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-26 patch sysinv 2020-05-20 16:15:26.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:29.677 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-29 patch sysinv 2020-05-20 16:15:29.678 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:29.710 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-29 patch sysinv 2020-05-20 16:15:29.757 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-29 patch sysinv 2020-05-20 16:15:29.757 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:29.809 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-29 patch sysinv 2020-05-20 16:15:29.809 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:29.870 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-29 patch sysinv 2020-05-20 16:15:36.145 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-36 patch sysinv 2020-05-20 16:15:36.146 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:36.189 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-36 patch sysinv 2020-05-20 16:15:36.190 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:40.156 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:15:40.157 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:15:40.158 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:15:40.158 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:15:40.159 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:15:41.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-41 patch sysinv 2020-05-20 16:15:41.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:41.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-41 patch sysinv 2020-05-20 16:15:41.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-41 patch sysinv 2020-05-20 16:15:41.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:41.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-41 patch sysinv 2020-05-20 16:15:41.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:41.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-41 patch sysinv 2020-05-20 16:15:44.175 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-44 patch sysinv 2020-05-20 16:15:44.175 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:44.194 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-44 patch sysinv 2020-05-20 16:15:44.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-44 patch sysinv 2020-05-20 16:15:44.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:44.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-44 patch sysinv 2020-05-20 16:15:44.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:15:44.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-15-44 patch sysinv 2020-05-20 16:15:46.162 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-46 patch sysinv 2020-05-20 16:15:46.163 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:46.210 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-46 patch sysinv 2020-05-20 16:15:46.210 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:56.185 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-56 patch sysinv 2020-05-20 16:15:56.185 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:15:56.234 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-15-56 patch sysinv 2020-05-20 16:15:56.234 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:03.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-03 patch sysinv 2020-05-20 16:16:03.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:03.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-03 patch sysinv 2020-05-20 16:16:03.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-03 patch sysinv 2020-05-20 16:16:03.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:03.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-03 patch sysinv 2020-05-20 16:16:03.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:03.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-03 patch sysinv 2020-05-20 16:16:04.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-04 patch sysinv 2020-05-20 16:16:04.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:04.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-04 patch sysinv 2020-05-20 16:16:04.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-04 patch sysinv 2020-05-20 16:16:04.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:04.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-04 patch sysinv 2020-05-20 16:16:04.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:04.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-04 patch sysinv 2020-05-20 16:16:06.219 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-06 patch sysinv 2020-05-20 16:16:06.220 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:06.276 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-06 patch sysinv 2020-05-20 16:16:06.276 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:15.274 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-15 patch sysinv 2020-05-20 16:16:15.275 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:15.292 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-15 patch sysinv 2020-05-20 16:16:15.337 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-15 patch sysinv 2020-05-20 16:16:15.337 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:15.387 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-15 patch sysinv 2020-05-20 16:16:15.388 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:15.444 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-15 patch sysinv 2020-05-20 16:16:16.225 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-16 patch sysinv 2020-05-20 16:16:16.225 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:16.276 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-16 patch sysinv 2020-05-20 16:16:16.276 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:25.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-25 patch sysinv 2020-05-20 16:16:25.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:25.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-25 patch sysinv 2020-05-20 16:16:25.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-25 patch sysinv 2020-05-20 16:16:25.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:25.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-25 patch sysinv 2020-05-20 16:16:25.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:25.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-25 patch sysinv 2020-05-20 16:16:26.245 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-26 patch sysinv 2020-05-20 16:16:26.245 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:26.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-26 patch sysinv 2020-05-20 16:16:26.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:36.267 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-36 patch sysinv 2020-05-20 16:16:36.268 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:36.319 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-36 patch sysinv 2020-05-20 16:16:36.319 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:37.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-37 patch sysinv 2020-05-20 16:16:37.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:37.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-37 patch sysinv 2020-05-20 16:16:37.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-37 patch sysinv 2020-05-20 16:16:37.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:37.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-37 patch sysinv 2020-05-20 16:16:37.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:37.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-37 patch sysinv 2020-05-20 16:16:40.242 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:16:40.243 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:16:40.244 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:16:40.244 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:16:40.244 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:16:46.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:46.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:46.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:46.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.410 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:46.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-46 patch sysinv 2020-05-20 16:16:46.462 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:49.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-49 patch sysinv 2020-05-20 16:16:49.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:49.201 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-49 patch sysinv 2020-05-20 16:16:49.258 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-49 patch sysinv 2020-05-20 16:16:49.259 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:49.310 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-49 patch sysinv 2020-05-20 16:16:49.311 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:16:49.371 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-16-49 patch sysinv 2020-05-20 16:16:56.303 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-56 patch sysinv 2020-05-20 16:16:56.304 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:16:56.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-16-56 patch sysinv 2020-05-20 16:16:56.357 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:06.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-06 patch sysinv 2020-05-20 16:17:06.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:06.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-06 patch sysinv 2020-05-20 16:17:06.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:07.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-07 patch sysinv 2020-05-20 16:17:07.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:07.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-07 patch sysinv 2020-05-20 16:17:07.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-07 patch sysinv 2020-05-20 16:17:07.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:07.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-07 patch sysinv 2020-05-20 16:17:07.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:07.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-07 patch sysinv 2020-05-20 16:17:11.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-11 patch sysinv 2020-05-20 16:17:11.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:11.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-11 patch sysinv 2020-05-20 16:17:11.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-11 patch sysinv 2020-05-20 16:17:11.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:11.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-11 patch sysinv 2020-05-20 16:17:11.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:11.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-11 patch sysinv 2020-05-20 16:17:16.351 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-16 patch sysinv 2020-05-20 16:17:16.352 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:16.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-16 patch sysinv 2020-05-20 16:17:16.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:22.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-22 patch sysinv 2020-05-20 16:17:22.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:22.695 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-22 patch sysinv 2020-05-20 16:17:22.744 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-22 patch sysinv 2020-05-20 16:17:22.745 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:22.796 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-22 patch sysinv 2020-05-20 16:17:22.796 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:22.851 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-22 patch sysinv 2020-05-20 16:17:26.366 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-26 patch sysinv 2020-05-20 16:17:26.367 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:26.418 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-26 patch sysinv 2020-05-20 16:17:26.418 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:27.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-27 patch sysinv 2020-05-20 16:17:27.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:27.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-27 patch sysinv 2020-05-20 16:17:27.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-27 patch sysinv 2020-05-20 16:17:27.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:27.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-27 patch sysinv 2020-05-20 16:17:27.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:27.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-27 patch sysinv 2020-05-20 16:17:36.387 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-36 patch sysinv 2020-05-20 16:17:36.387 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:36.434 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-36 patch sysinv 2020-05-20 16:17:36.435 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:40.226 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:17:40.227 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:17:40.228 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:17:40.229 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:17:40.229 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:17:45.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-45 patch sysinv 2020-05-20 16:17:45.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:45.200 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-45 patch sysinv 2020-05-20 16:17:45.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-45 patch sysinv 2020-05-20 16:17:45.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:45.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-45 patch sysinv 2020-05-20 16:17:45.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:45.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-45 patch sysinv 2020-05-20 16:17:46.408 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-46 patch sysinv 2020-05-20 16:17:46.408 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:46.458 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-46 patch sysinv 2020-05-20 16:17:46.459 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:47.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-47 patch sysinv 2020-05-20 16:17:47.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:47.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-47 patch sysinv 2020-05-20 16:17:47.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-47 patch sysinv 2020-05-20 16:17:47.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:47.392 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-47 patch sysinv 2020-05-20 16:17:47.393 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:47.452 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-47 patch sysinv 2020-05-20 16:17:56.432 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.433 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:56.478 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.478 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:56.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:56.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:17:56.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:17:56.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:17:56.846 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-17-56 patch sysinv 2020-05-20 16:18:06.465 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-06 patch sysinv 2020-05-20 16:18:06.465 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:06.518 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-06 patch sysinv 2020-05-20 16:18:06.518 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:08.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-08 patch sysinv 2020-05-20 16:18:08.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:08.193 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-08 patch sysinv 2020-05-20 16:18:08.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-08 patch sysinv 2020-05-20 16:18:08.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:08.285 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-08 patch sysinv 2020-05-20 16:18:08.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:08.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-08 patch sysinv 2020-05-20 16:18:16.475 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-16 patch sysinv 2020-05-20 16:18:16.476 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:16.518 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-16 patch sysinv 2020-05-20 16:18:16.518 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:19.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-19 patch sysinv 2020-05-20 16:18:19.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:19.186 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-19 patch sysinv 2020-05-20 16:18:19.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-19 patch sysinv 2020-05-20 16:18:19.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:19.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-19 patch sysinv 2020-05-20 16:18:19.294 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:19.360 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-19 patch sysinv 2020-05-20 16:18:26.492 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-26 patch sysinv 2020-05-20 16:18:26.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:26.542 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-26 patch sysinv 2020-05-20 16:18:26.542 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:29.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-29 patch sysinv 2020-05-20 16:18:29.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:29.200 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-29 patch sysinv 2020-05-20 16:18:29.245 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-29 patch sysinv 2020-05-20 16:18:29.246 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:29.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-29 patch sysinv 2020-05-20 16:18:29.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:29.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-29 patch sysinv 2020-05-20 16:18:31.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-31 patch sysinv 2020-05-20 16:18:31.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:31.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-31 patch sysinv 2020-05-20 16:18:31.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-31 patch sysinv 2020-05-20 16:18:31.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:31.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-31 patch sysinv 2020-05-20 16:18:31.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:31.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-31 patch sysinv 2020-05-20 16:18:36.513 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-36 patch sysinv 2020-05-20 16:18:36.513 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:36.561 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-36 patch sysinv 2020-05-20 16:18:36.561 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:40.202 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:18:40.203 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:18:40.204 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:18:40.204 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:18:40.205 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:18:46.533 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-46 patch sysinv 2020-05-20 16:18:46.534 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:46.584 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-46 patch sysinv 2020-05-20 16:18:46.585 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:50.175 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-50 patch sysinv 2020-05-20 16:18:50.176 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:50.204 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-50 patch sysinv 2020-05-20 16:18:50.257 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-50 patch sysinv 2020-05-20 16:18:50.257 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:50.315 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-50 patch sysinv 2020-05-20 16:18:50.316 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:50.372 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-50 patch sysinv 2020-05-20 16:18:53.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-53 patch sysinv 2020-05-20 16:18:53.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:53.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-53 patch sysinv 2020-05-20 16:18:53.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-53 patch sysinv 2020-05-20 16:18:53.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:53.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-53 patch sysinv 2020-05-20 16:18:53.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:18:53.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-18-53 patch sysinv 2020-05-20 16:18:56.555 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-56 patch sysinv 2020-05-20 16:18:56.556 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:18:56.602 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-18-56 patch sysinv 2020-05-20 16:18:56.603 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:06.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-06 patch sysinv 2020-05-20 16:19:06.590 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:06.644 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-06 patch sysinv 2020-05-20 16:19:06.644 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:07.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-07 patch sysinv 2020-05-20 16:19:07.675 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:07.698 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-07 patch sysinv 2020-05-20 16:19:07.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-07 patch sysinv 2020-05-20 16:19:07.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:07.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-07 patch sysinv 2020-05-20 16:19:07.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:07.847 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-07 patch sysinv 2020-05-20 16:19:11.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-11 patch sysinv 2020-05-20 16:19:11.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:11.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-11 patch sysinv 2020-05-20 16:19:11.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-11 patch sysinv 2020-05-20 16:19:11.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:11.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-11 patch sysinv 2020-05-20 16:19:11.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:11.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-11 patch sysinv 2020-05-20 16:19:16.598 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-16 patch sysinv 2020-05-20 16:19:16.598 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:16.750 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-16 patch sysinv 2020-05-20 16:19:16.750 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:26.618 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-26 patch sysinv 2020-05-20 16:19:26.618 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:26.666 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-26 patch sysinv 2020-05-20 16:19:26.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:28.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-28 patch sysinv 2020-05-20 16:19:28.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:28.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-28 patch sysinv 2020-05-20 16:19:28.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-28 patch sysinv 2020-05-20 16:19:28.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:28.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-28 patch sysinv 2020-05-20 16:19:28.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:28.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-28 patch sysinv 2020-05-20 16:19:32.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-32 patch sysinv 2020-05-20 16:19:32.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:32.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-32 patch sysinv 2020-05-20 16:19:32.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-32 patch sysinv 2020-05-20 16:19:32.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:32.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-32 patch sysinv 2020-05-20 16:19:32.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:32.345 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-32 patch sysinv 2020-05-20 16:19:36.639 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-36 patch sysinv 2020-05-20 16:19:36.639 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:36.688 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-36 patch sysinv 2020-05-20 16:19:36.688 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:40.274 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:19:40.275 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:19:40.276 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:19:40.276 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:19:40.277 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:19:42.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-42 patch sysinv 2020-05-20 16:19:42.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:42.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-42 patch sysinv 2020-05-20 16:19:42.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-42 patch sysinv 2020-05-20 16:19:42.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-42 patch sysinv 2020-05-20 16:19:42.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:42.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-42 patch sysinv 2020-05-20 16:19:46.753 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-46 patch sysinv 2020-05-20 16:19:46.753 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:46.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-46 patch sysinv 2020-05-20 16:19:46.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:53.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-53 patch sysinv 2020-05-20 16:19:53.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:53.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-53 patch sysinv 2020-05-20 16:19:53.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-53 patch sysinv 2020-05-20 16:19:53.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:53.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-53 patch sysinv 2020-05-20 16:19:53.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:19:53.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-19-53 patch sysinv 2020-05-20 16:19:56.682 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-56 patch sysinv 2020-05-20 16:19:56.683 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:19:56.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-19-56 patch sysinv 2020-05-20 16:19:56.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:03.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-03 patch sysinv 2020-05-20 16:20:03.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:03.194 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-03 patch sysinv 2020-05-20 16:20:03.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-03 patch sysinv 2020-05-20 16:20:03.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:03.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-03 patch sysinv 2020-05-20 16:20:03.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:03.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-03 patch sysinv 2020-05-20 16:20:06.721 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-06 patch sysinv 2020-05-20 16:20:06.721 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:06.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-06 patch sysinv 2020-05-20 16:20:06.771 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:13.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-13 patch sysinv 2020-05-20 16:20:13.171 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:13.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-13 patch sysinv 2020-05-20 16:20:13.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-13 patch sysinv 2020-05-20 16:20:13.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:13.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-13 patch sysinv 2020-05-20 16:20:13.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:13.346 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-13 patch sysinv 2020-05-20 16:20:16.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:16.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.728 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:16.778 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.779 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:16.833 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.901 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:16.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-16 patch sysinv 2020-05-20 16:20:16.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:26.743 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-26 patch sysinv 2020-05-20 16:20:26.743 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:26.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-26 patch sysinv 2020-05-20 16:20:26.791 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:33.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-33 patch sysinv 2020-05-20 16:20:33.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:33.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-33 patch sysinv 2020-05-20 16:20:33.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-33 patch sysinv 2020-05-20 16:20:33.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:33.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-33 patch sysinv 2020-05-20 16:20:33.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:33.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-33 patch sysinv 2020-05-20 16:20:36.763 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-36 patch sysinv 2020-05-20 16:20:36.763 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:36.814 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-36 patch sysinv 2020-05-20 16:20:36.815 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:38.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-38 patch sysinv 2020-05-20 16:20:38.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:38.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-38 patch sysinv 2020-05-20 16:20:38.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-38 patch sysinv 2020-05-20 16:20:38.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:38.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-38 patch sysinv 2020-05-20 16:20:38.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:38.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-38 patch sysinv 2020-05-20 16:20:40.259 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:20:40.260 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:20:40.261 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:20:40.262 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:20:40.262 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:20:46.785 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-46 patch sysinv 2020-05-20 16:20:46.785 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:46.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-46 patch sysinv 2020-05-20 16:20:46.833 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:50.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-50 patch sysinv 2020-05-20 16:20:50.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:50.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-50 patch sysinv 2020-05-20 16:20:50.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-50 patch sysinv 2020-05-20 16:20:50.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:50.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-50 patch sysinv 2020-05-20 16:20:50.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:50.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-50 patch sysinv 2020-05-20 16:20:54.166 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-54 patch sysinv 2020-05-20 16:20:54.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:54.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-54 patch sysinv 2020-05-20 16:20:54.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-54 patch sysinv 2020-05-20 16:20:54.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:54.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-54 patch sysinv 2020-05-20 16:20:54.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:20:54.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-20-54 patch sysinv 2020-05-20 16:20:56.920 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-56 patch sysinv 2020-05-20 16:20:56.920 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:20:56.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-20-56 patch sysinv 2020-05-20 16:20:56.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:06.839 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-06 patch sysinv 2020-05-20 16:21:06.840 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:06.875 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-06 patch sysinv 2020-05-20 16:21:06.875 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:13.175 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-13 patch sysinv 2020-05-20 16:21:13.176 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:13.202 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-13 patch sysinv 2020-05-20 16:21:13.258 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-13 patch sysinv 2020-05-20 16:21:13.258 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:13.309 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-13 patch sysinv 2020-05-20 16:21:13.309 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:13.371 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-13 patch sysinv 2020-05-20 16:21:14.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-14 patch sysinv 2020-05-20 16:21:14.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:14.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-14 patch sysinv 2020-05-20 16:21:14.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-14 patch sysinv 2020-05-20 16:21:14.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:14.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-14 patch sysinv 2020-05-20 16:21:14.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:14.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-14 patch sysinv 2020-05-20 16:21:16.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-16 patch sysinv 2020-05-20 16:21:16.850 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:16.896 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-16 patch sysinv 2020-05-20 16:21:16.897 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:26.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:26.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:26.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:26.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.873 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.873 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:26.916 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-26 patch sysinv 2020-05-20 16:21:26.917 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:35.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-35 patch sysinv 2020-05-20 16:21:35.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:35.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-35 patch sysinv 2020-05-20 16:21:35.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-35 patch sysinv 2020-05-20 16:21:35.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:35.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-35 patch sysinv 2020-05-20 16:21:35.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:35.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-35 patch sysinv 2020-05-20 16:21:36.892 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-36 patch sysinv 2020-05-20 16:21:36.892 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:36.936 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-36 patch sysinv 2020-05-20 16:21:36.937 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:40.341 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:21:40.342 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:21:40.342 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:21:40.343 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:21:40.343 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:21:46.910 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-46 patch sysinv 2020-05-20 16:21:46.911 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:46.965 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-46 patch sysinv 2020-05-20 16:21:46.966 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:47.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-47 patch sysinv 2020-05-20 16:21:47.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:47.695 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-47 patch sysinv 2020-05-20 16:21:47.733 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-47 patch sysinv 2020-05-20 16:21:47.734 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:47.784 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-47 patch sysinv 2020-05-20 16:21:47.784 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:47.839 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-47 patch sysinv 2020-05-20 16:21:56.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.172 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:56.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:56.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:56.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.935 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.935 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:56.976 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-56 patch sysinv 2020-05-20 16:21:56.977 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:59.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-59 patch sysinv 2020-05-20 16:21:59.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:59.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-59 patch sysinv 2020-05-20 16:21:59.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-59 patch sysinv 2020-05-20 16:21:59.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:21:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-21-59 patch sysinv 2020-05-20 16:21:59.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:21:59.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-21-59 patch sysinv 2020-05-20 16:22:06.966 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-06 patch sysinv 2020-05-20 16:22:06.967 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:07.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-07 patch sysinv 2020-05-20 16:22:07.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:16.973 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-16 patch sysinv 2020-05-20 16:22:16.974 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:17.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:17.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:17.164 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:17.165 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:17.182 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:17.216 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:17.217 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:17.265 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:17.265 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:17.322 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-17 patch sysinv 2020-05-20 16:22:21.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-21 patch sysinv 2020-05-20 16:22:21.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:21.185 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-21 patch sysinv 2020-05-20 16:22:21.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-21 patch sysinv 2020-05-20 16:22:21.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:21.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-21 patch sysinv 2020-05-20 16:22:21.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:21.443 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-21 patch sysinv 2020-05-20 16:22:26.992 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-26 patch sysinv 2020-05-20 16:22:26.993 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:27.038 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-27 patch sysinv 2020-05-20 16:22:27.039 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:33.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-33 patch sysinv 2020-05-20 16:22:33.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:33.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-33 patch sysinv 2020-05-20 16:22:33.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-33 patch sysinv 2020-05-20 16:22:33.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:33.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-33 patch sysinv 2020-05-20 16:22:33.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:33.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-33 patch sysinv 2020-05-20 16:22:37.011 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-36 patch sysinv 2020-05-20 16:22:37.011 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:37.062 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:37.063 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:37.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:37.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:37.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:37.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:37.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:37.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:37.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:37.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-37 patch sysinv 2020-05-20 16:22:40.318 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:22:40.319 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:22:40.319 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:22:40.320 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:22:40.320 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:22:47.041 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-47 patch sysinv 2020-05-20 16:22:47.042 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:47.080 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-47 patch sysinv 2020-05-20 16:22:47.080 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:56.674 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-56 patch sysinv 2020-05-20 16:22:56.675 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:56.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-56 patch sysinv 2020-05-20 16:22:56.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-56 patch sysinv 2020-05-20 16:22:56.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:56.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-56 patch sysinv 2020-05-20 16:22:56.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:56.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-56 patch sysinv 2020-05-20 16:22:57.050 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-57 patch sysinv 2020-05-20 16:22:57.050 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:57.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-57 patch sysinv 2020-05-20 16:22:57.100 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:58.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-58 patch sysinv 2020-05-20 16:22:58.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:58.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-58 patch sysinv 2020-05-20 16:22:58.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-58 patch sysinv 2020-05-20 16:22:58.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:22:58.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-22-58 patch sysinv 2020-05-20 16:22:58.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:22:58.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-22-58 patch sysinv 2020-05-20 16:23:07.088 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-07 patch sysinv 2020-05-20 16:23:07.088 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:07.124 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-07 patch sysinv 2020-05-20 16:23:07.124 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:08.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-08 patch sysinv 2020-05-20 16:23:08.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:08.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-08 patch sysinv 2020-05-20 16:23:08.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-08 patch sysinv 2020-05-20 16:23:08.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:08.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-08 patch sysinv 2020-05-20 16:23:08.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:08.348 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-08 patch sysinv 2020-05-20 16:23:17.095 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-17 patch sysinv 2020-05-20 16:23:17.095 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:17.142 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-17 patch sysinv 2020-05-20 16:23:17.142 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:19.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-19 patch sysinv 2020-05-20 16:23:19.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:19.695 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-19 patch sysinv 2020-05-20 16:23:19.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-19 patch sysinv 2020-05-20 16:23:19.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:19.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-19 patch sysinv 2020-05-20 16:23:19.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:19.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-19 patch sysinv 2020-05-20 16:23:27.116 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-27 patch sysinv 2020-05-20 16:23:27.116 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:27.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-27 patch sysinv 2020-05-20 16:23:27.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:30.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-30 patch sysinv 2020-05-20 16:23:30.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:30.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-30 patch sysinv 2020-05-20 16:23:30.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-30 patch sysinv 2020-05-20 16:23:30.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:30.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-30 patch sysinv 2020-05-20 16:23:30.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:30.849 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-30 patch sysinv 2020-05-20 16:23:37.137 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-37 patch sysinv 2020-05-20 16:23:37.137 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:37.184 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-37 patch sysinv 2020-05-20 16:23:37.185 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:40.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-40 patch sysinv 2020-05-20 16:23:40.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:40.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-40 patch sysinv 2020-05-20 16:23:40.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-40 patch sysinv 2020-05-20 16:23:40.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:40.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-40 patch sysinv 2020-05-20 16:23:40.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:40.342 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-40 patch sysinv 2020-05-20 16:23:40.402 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:23:40.403 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:23:40.403 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:23:40.404 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:23:40.404 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:23:42.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-42 patch sysinv 2020-05-20 16:23:42.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:42.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-42 patch sysinv 2020-05-20 16:23:42.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-42 patch sysinv 2020-05-20 16:23:42.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:42.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-42 patch sysinv 2020-05-20 16:23:42.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:23:42.841 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-23-42 patch sysinv 2020-05-20 16:23:47.158 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-47 patch sysinv 2020-05-20 16:23:47.159 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:47.209 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-47 patch sysinv 2020-05-20 16:23:47.209 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:57.179 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-57 patch sysinv 2020-05-20 16:23:57.179 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:23:57.226 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-23-57 patch sysinv 2020-05-20 16:23:57.226 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:01.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-01 patch sysinv 2020-05-20 16:24:01.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:01.293 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-01 patch sysinv 2020-05-20 16:24:01.336 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-01 patch sysinv 2020-05-20 16:24:01.336 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:01.388 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-01 patch sysinv 2020-05-20 16:24:01.388 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:01.444 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-01 patch sysinv 2020-05-20 16:24:04.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-04 patch sysinv 2020-05-20 16:24:04.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:04.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-04 patch sysinv 2020-05-20 16:24:04.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-04 patch sysinv 2020-05-20 16:24:04.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:04.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-04 patch sysinv 2020-05-20 16:24:04.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:04.847 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-04 patch sysinv 2020-05-20 16:24:07.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-07 patch sysinv 2020-05-20 16:24:07.215 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:07.269 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-07 patch sysinv 2020-05-20 16:24:07.269 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:17.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:17.191 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.227 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.227 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:17.277 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.278 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:17.335 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.400 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.400 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:17.453 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-17 patch sysinv 2020-05-20 16:24:17.453 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:21.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-21 patch sysinv 2020-05-20 16:24:21.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:21.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-21 patch sysinv 2020-05-20 16:24:21.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-21 patch sysinv 2020-05-20 16:24:21.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:21.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-21 patch sysinv 2020-05-20 16:24:21.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:21.344 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-21 patch sysinv 2020-05-20 16:24:27.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-27 patch sysinv 2020-05-20 16:24:27.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:27.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-27 patch sysinv 2020-05-20 16:24:27.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:37.261 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-37 patch sysinv 2020-05-20 16:24:37.261 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:37.312 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-37 patch sysinv 2020-05-20 16:24:37.313 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:38.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-38 patch sysinv 2020-05-20 16:24:38.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:38.688 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-38 patch sysinv 2020-05-20 16:24:38.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-38 patch sysinv 2020-05-20 16:24:38.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:38.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-38 patch sysinv 2020-05-20 16:24:38.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:38.842 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-38 patch sysinv 2020-05-20 16:24:40.383 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:24:40.384 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:24:40.385 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:24:40.385 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:24:40.386 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:24:42.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-42 patch sysinv 2020-05-20 16:24:42.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:42.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-42 patch sysinv 2020-05-20 16:24:42.240 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-42 patch sysinv 2020-05-20 16:24:42.241 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:42.285 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-42 patch sysinv 2020-05-20 16:24:42.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:42.359 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-42 patch sysinv 2020-05-20 16:24:47.312 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-47 patch sysinv 2020-05-20 16:24:47.313 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:47.352 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-47 patch sysinv 2020-05-20 16:24:47.352 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:51.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-51 patch sysinv 2020-05-20 16:24:51.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:51.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-51 patch sysinv 2020-05-20 16:24:51.735 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-51 patch sysinv 2020-05-20 16:24:51.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:51.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-51 patch sysinv 2020-05-20 16:24:51.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:24:51.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-24-51 patch sysinv 2020-05-20 16:24:57.303 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-57 patch sysinv 2020-05-20 16:24:57.304 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:24:57.350 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-24-57 patch sysinv 2020-05-20 16:24:57.350 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:03.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-03 patch sysinv 2020-05-20 16:25:03.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:03.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-03 patch sysinv 2020-05-20 16:25:03.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-03 patch sysinv 2020-05-20 16:25:03.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:03.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-03 patch sysinv 2020-05-20 16:25:03.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:03.352 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-03 patch sysinv 2020-05-20 16:25:07.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-07 patch sysinv 2020-05-20 16:25:07.339 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:07.395 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-07 patch sysinv 2020-05-20 16:25:07.395 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:14.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-14 patch sysinv 2020-05-20 16:25:14.170 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:14.190 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-14 patch sysinv 2020-05-20 16:25:14.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-14 patch sysinv 2020-05-20 16:25:14.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:14.289 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-14 patch sysinv 2020-05-20 16:25:14.290 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:14.347 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-14 patch sysinv 2020-05-20 16:25:17.351 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-17 patch sysinv 2020-05-20 16:25:17.352 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:17.392 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-17 patch sysinv 2020-05-20 16:25:17.392 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:23.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-23 patch sysinv 2020-05-20 16:25:23.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:23.693 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-23 patch sysinv 2020-05-20 16:25:23.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-23 patch sysinv 2020-05-20 16:25:23.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:23.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-23 patch sysinv 2020-05-20 16:25:23.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:23.843 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-23 patch sysinv 2020-05-20 16:25:26.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-26 patch sysinv 2020-05-20 16:25:26.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:26.188 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-26 patch sysinv 2020-05-20 16:25:26.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-26 patch sysinv 2020-05-20 16:25:26.235 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:26.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-26 patch sysinv 2020-05-20 16:25:26.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:26.341 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-26 patch sysinv 2020-05-20 16:25:27.366 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-27 patch sysinv 2020-05-20 16:25:27.366 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:27.522 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-27 patch sysinv 2020-05-20 16:25:27.523 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:37.384 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-37 patch sysinv 2020-05-20 16:25:37.385 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:37.433 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-37 patch sysinv 2020-05-20 16:25:37.433 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:40.456 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:25:40.457 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:25:40.457 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:25:40.458 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:25:40.458 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:25:44.183 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-44 patch sysinv 2020-05-20 16:25:44.183 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:44.209 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-44 patch sysinv 2020-05-20 16:25:44.262 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-44 patch sysinv 2020-05-20 16:25:44.262 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:44.308 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-44 patch sysinv 2020-05-20 16:25:44.308 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:44.370 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-44 patch sysinv 2020-05-20 16:25:47.407 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-47 patch sysinv 2020-05-20 16:25:47.407 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:47.455 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-47 patch sysinv 2020-05-20 16:25:47.455 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:48.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-48 patch sysinv 2020-05-20 16:25:48.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:48.189 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-48 patch sysinv 2020-05-20 16:25:48.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-48 patch sysinv 2020-05-20 16:25:48.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:48.286 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-48 patch sysinv 2020-05-20 16:25:48.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:25:48.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-25-48 patch sysinv 2020-05-20 16:25:57.429 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-57 patch sysinv 2020-05-20 16:25:57.429 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:25:57.479 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-25-57 patch sysinv 2020-05-20 16:25:57.480 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:01.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-01 patch sysinv 2020-05-20 16:26:01.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:01.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-01 patch sysinv 2020-05-20 16:26:01.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-01 patch sysinv 2020-05-20 16:26:01.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:01.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-01 patch sysinv 2020-05-20 16:26:01.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:01.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-01 patch sysinv 2020-05-20 16:26:04.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-04 patch sysinv 2020-05-20 16:26:04.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:04.691 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-04 patch sysinv 2020-05-20 16:26:04.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-04 patch sysinv 2020-05-20 16:26:04.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:04.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-04 patch sysinv 2020-05-20 16:26:04.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:04.847 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-04 patch sysinv 2020-05-20 16:26:07.464 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-07 patch sysinv 2020-05-20 16:26:07.465 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:07.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-07 patch sysinv 2020-05-20 16:26:07.517 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:17.470 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-17 patch sysinv 2020-05-20 16:26:17.470 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:17.521 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-17 patch sysinv 2020-05-20 16:26:17.522 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:22.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-22 patch sysinv 2020-05-20 16:26:22.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:22.689 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-22 patch sysinv 2020-05-20 16:26:22.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-22 patch sysinv 2020-05-20 16:26:22.738 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:22.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-22 patch sysinv 2020-05-20 16:26:22.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:22.844 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-22 patch sysinv 2020-05-20 16:26:25.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-25 patch sysinv 2020-05-20 16:26:25.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:25.690 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-25 patch sysinv 2020-05-20 16:26:25.740 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-25 patch sysinv 2020-05-20 16:26:25.740 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:25.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-25 patch sysinv 2020-05-20 16:26:25.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:25.847 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-25 patch sysinv 2020-05-20 16:26:27.492 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-27 patch sysinv 2020-05-20 16:26:27.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:27.540 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-27 patch sysinv 2020-05-20 16:26:27.541 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:35.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-35 patch sysinv 2020-05-20 16:26:35.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:35.692 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-35 patch sysinv 2020-05-20 16:26:35.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-35 patch sysinv 2020-05-20 16:26:35.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:35.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-35 patch sysinv 2020-05-20 16:26:35.790 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:35.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-35 patch sysinv 2020-05-20 16:26:37.512 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-37 patch sysinv 2020-05-20 16:26:37.513 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:37.561 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-37 patch sysinv 2020-05-20 16:26:37.561 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:40.441 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:26:40.442 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:26:40.443 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:26:40.443 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:26:40.444 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:26:46.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-46 patch sysinv 2020-05-20 16:26:46.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:46.694 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-46 patch sysinv 2020-05-20 16:26:46.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-46 patch sysinv 2020-05-20 16:26:46.737 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:46.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-46 patch sysinv 2020-05-20 16:26:46.789 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:46.845 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-46 patch sysinv 2020-05-20 16:26:47.531 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-47 patch sysinv 2020-05-20 16:26:47.532 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:47.583 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-47 patch sysinv 2020-05-20 16:26:47.583 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:57.552 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-57 patch sysinv 2020-05-20 16:26:57.552 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:57.605 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-57 patch sysinv 2020-05-20 16:26:57.606 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:58.173 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-58 patch sysinv 2020-05-20 16:26:58.174 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:58.195 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-58 patch sysinv 2020-05-20 16:26:58.236 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-58 patch sysinv 2020-05-20 16:26:58.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:26:58.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-26-58 patch sysinv 2020-05-20 16:26:58.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:26:58.354 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-26-58 patch sysinv 2020-05-20 16:27:07.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.169 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:07.192 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:07.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.288 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:07.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.588 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.589 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:07.621 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-07 patch sysinv 2020-05-20 16:27:07.622 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:10.667 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-10 patch sysinv 2020-05-20 16:27:10.668 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:10.687 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-10 patch sysinv 2020-05-20 16:27:10.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-10 patch sysinv 2020-05-20 16:27:10.741 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:10.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-10 patch sysinv 2020-05-20 16:27:10.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:10.846 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-10 patch sysinv 2020-05-20 16:27:17.596 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-17 patch sysinv 2020-05-20 16:27:17.597 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:17.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-17 patch sysinv 2020-05-20 16:27:17.643 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:27.620 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-27 patch sysinv 2020-05-20 16:27:27.621 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:27.772 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-27 patch sysinv 2020-05-20 16:27:27.773 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:28.167 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-28 patch sysinv 2020-05-20 16:27:28.168 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:28.187 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-28 patch sysinv 2020-05-20 16:27:28.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-28 patch sysinv 2020-05-20 16:27:28.237 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:28.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-28 patch sysinv 2020-05-20 16:27:28.287 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:28.343 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-28 patch sysinv 2020-05-20 16:27:32.179 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-32 patch sysinv 2020-05-20 16:27:32.179 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:32.199 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-32 patch sysinv 2020-05-20 16:27:32.238 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-32 patch sysinv 2020-05-20 16:27:32.239 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:32.292 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-32 patch sysinv 2020-05-20 16:27:32.293 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:32.363 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-32 patch sysinv 2020-05-20 16:27:37.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-37 patch sysinv 2020-05-20 16:27:37.641 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:37.687 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-37 patch sysinv 2020-05-20 16:27:37.687 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:40.529 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:27:40.529 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:27:40.530 90642 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:27:40.531 90642 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-20 16:27:40.531 90642 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 16:27:43.669 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-43 patch sysinv 2020-05-20 16:27:43.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:43.701 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-43 patch sysinv 2020-05-20 16:27:43.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-43 patch sysinv 2020-05-20 16:27:43.736 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:43.786 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-43 patch sysinv 2020-05-20 16:27:43.787 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:43.862 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-43 patch sysinv 2020-05-20 16:27:47.659 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-47 patch sysinv 2020-05-20 16:27:47.659 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:47.711 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-47 patch sysinv 2020-05-20 16:27:47.711 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:48.670 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-48 patch sysinv 2020-05-20 16:27:48.671 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:48.697 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-48 patch sysinv 2020-05-20 16:27:48.739 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-48 patch sysinv 2020-05-20 16:27:48.740 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:27:48.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-48 patch sysinv 2020-05-20 16:27:48.788 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:48.846 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-48 patch sysinv 2020-05-20 16:27:49.928 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-49 patch sysinv 2020-05-20 16:27:49.928 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-20 16:27:49.929 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock_controller sysinv 2020-05-20 16:27:49.954 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action=force-lock ihost_val_prenotify: {'ihost_action': 'force-lock'} ihost_val: {'ihost_action': 'force-lock', 'task': u'Force Locking'} sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'force-lock'} sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=force-lock, notify_vim=True notify_mtce=False rc=True sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=False sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action force-lock sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_force_lock_action sysinv 2020-05-20 16:27:49.955 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: force-lock sysinv 2020-05-20 16:27:49.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=True sysinv 2020-05-20 16:27:49.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-20 16:27:49.956 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=True sysinv 2020-05-20 16:27:49.956 91235 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'force-lock'} sysinv 2020-05-20 16:27:49.965 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'force-lock', 'task': u'Force Locking'} sysinv 2020-05-20 16:27:49.965 91235 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=force-lock sysinv 2020-05-20 16:27:49.965 91235 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=force-lock sysinv 2020-05-20 16:27:49.966 91235 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=force-lock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'force-lock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-20 16:27:49.966 91235 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "force-lock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-20 16:27:49.974 91235 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-20 16:27:50.029 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-50 patch sysinv 2020-05-20 16:27:55.015 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-54 patch sysinv 2020-05-20 16:27:55.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-20 16:27:55.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disable-failed sysinv 2020-05-20 16:27:55.016 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disable-failed ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-20 16:27:55.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disable-failed, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-20 16:27:55.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:27:55.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disable-failed sysinv 2020-05-20 16:27:55.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 handle_vim_services_disable_failed ihost_action=force-lock sysinv 2020-05-20 16:27:55.017 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-disable-failed sysinv 2020-05-20 16:27:55.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:27:55.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-20 16:27:55.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:27:55.018 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-disable-failed'} sysinv 2020-05-20 16:27:55.028 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 Action services-disable-failed perform notify_mtce sysinv 2020-05-20 16:27:55.029 91235 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-20 16:27:55.029 91235 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "degraded", "invprovision": "provisioned", "peer_id": null, "administrative": "unlocked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "target_load": "19.12", "vsc_controllers": null, "operational": "enabled", "subfunction_avail": "failed", "action": "force-lock", "bm_type": "none"} sysinv 2020-05-20 16:27:55.040 91235 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-20 16:27:55.094 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-55 patch sysinv 2020-05-20 16:27:55.120 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-55 patch sysinv 2020-05-20 16:27:55.120 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:27:55.143 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-55 patch sysinv 2020-05-20 16:27:55.193 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-27-55 patch sysinv 2020-05-20 16:27:55.194 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-20 16:27:55.234 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-27-55 patch sysinv 2020-05-20 16:27:55.323 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:27:55.429 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:28:00.919 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-28-00 patch sysinv 2020-05-20 16:28:00.920 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-20 16:28:00.946 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-28-00 patch sysinv 2020-05-20 16:28:01.051 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-28-01 patch sysinv 2020-05-20 16:28:01.051 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['administrative'] sysinv 2020-05-20 16:28:01.051 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['administrative'] sysinv 2020-05-20 16:28:01.052 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-20 16:28:01.076 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-28-01 patch sysinv 2020-05-20 16:28:01.077 91235 INFO sysinv.api.controllers.v1.host [-] Update host memory for (controller-0) sysinv 2020-05-20 16:28:01.077 91235 INFO sysinv.conductor.rpcapi [-] ConductorApi.update_host_memory: sending host memory update request to conductor sysinv 2020-05-20 16:28:01.144 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-28-01 patch sysinv 2020-05-20 16:28:01.144 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-20 16:28:01.172 91235 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-28-01 patch sysinv 2020-05-20 16:28:01.493 91235 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:33:29.596 10083 INFO sysinv.agent.lldp.manager [-] Configured sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:33:29.599 10083 INFO sysinv.agent.lldp.manager [-] Loaded sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:33:29.599 10083 INFO sysinv.agent.lldp.manager [-] Registered sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:33:29.651 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:33:30.653 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:33:30.659 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:33:33.663 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:33:33.669 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:33:38.672 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:33:38.678 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:33:45.680 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:33:45.686 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:33:54.691 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:33:54.697 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:34:05.703 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:34:05.709 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:34:18.710 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:34:18.716 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:34:33.726 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:34:33.732 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:34:50.745 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:34:50.751 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:09.765 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:09.771 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:30.775 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:30.781 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:39.943 90600 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. sysinv 2020-05-20 16:35:39.944 90600 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". sysinv 2020-05-20 16:35:39.950 90600 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-20 16:35:39.954 90600 INFO oslo.service.wsgi [-] sysinv_api listening on 192.168.204.1:6385 sysinv 2020-05-20 16:35:39.954 90600 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-20 16:35:39.963 90600 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.963 90600 WARNING keystonemiddleware._common.config [-] The option "auth_url" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.964 90600 WARNING keystonemiddleware._common.config [-] The option "project_domain_name" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.964 90600 WARNING keystonemiddleware._common.config [-] The option "username" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.964 90600 WARNING keystonemiddleware._common.config [-] The option "project_name" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.964 90600 WARNING keystonemiddleware._common.config [-] The option "password" in conf is not known to auth_token sysinv 2020-05-20 16:35:39.966 90600 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-20 16:35:39.969 90600 INFO oslo.service.wsgi [-] sysinv_api_pxe listening on 169.254.202.1:6385 sysinv 2020-05-20 16:35:39.969 90600 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-20 16:35:40.002 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.002 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.003 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.004 90444 ERROR sysinv.common.kubernetes [-] Failed to delete custom object, Namespace kube-system: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:35:40.004 90444 WARNING sysinv.conductor.kube_app [-] Failed to clear Armada locks.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:35:40.054 90444 INFO sysinv.conductor.manager [-] sysinv-conductor start committed system={'system_mode': u'simplex', 'created_at': datetime.datetime(2020, 4, 21, 9, 57, 41, 963891, tzinfo=), 'uuid': u'1004aa49-f471-4897-97d4-8ba085382f03', 'software_version': u'19.12', 'service_project_name': u'services', 'system_type': u'All-in-one', 'name': u'3609e439-df7e-4046-bcd7-dabd81fb2f18', 'description': None, 'location': None, 'updated_at': datetime.datetime(2020, 4, 21, 9, 59, 7, 193953, tzinfo=), 'capabilities': {u'https_enabled': False, u'vswitch_type': u'none', u'region_config': False, u'sdn_enabled': False, u'shared_services': u'[]'}, 'id': 1, 'contact': None, 'security_feature': u'nopti nospectre_v2', 'services': 72, 'timezone': u'UTC', 'security_profile': u'standard', 'distributed_cloud_role': None, 'region_name': u'RegionOne'} sysinv 2020-05-20 16:35:40.080 90444 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:40.087 90444 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-20 16:35:40.114 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.118 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.119 90444 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:35:40.120 90444 ERROR sysinv.conductor.manager [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)). Failed to upgrade/downgrade tiller.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:35:53.800 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:53.808 10083 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:53.816 10083 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-20 16:35:53.817 10083 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. sysinv 2020-05-20 16:35:54.549 10083 WARNING sysinv.agent.pci [-] Enabling device enp1s0f1 to query link speed sysinv 2020-05-20 16:35:54.555 10083 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp1s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:54.556 10083 WARNING sysinv.agent.pci [-] Disabling device enp1s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:54.822 91045 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. sysinv 2020-05-20 16:35:54.901 10083 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f0 sysinv 2020-05-20 16:35:55.235 10083 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f1 sysinv 2020-05-20 16:35:55.569 10083 WARNING sysinv.agent.pci [-] Enabling device enp134s0f0 to query link speed sysinv 2020-05-20 16:35:55.575 10083 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f0 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:55.576 10083 WARNING sysinv.agent.pci [-] Disabling device enp134s0f0 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:55.909 10083 WARNING sysinv.agent.pci [-] Enabling device enp134s0f1 to query link speed sysinv 2020-05-20 16:35:56.091 10083 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:56.092 10083 WARNING sysinv.agent.pci [-] Disabling device enp134s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:35:56.160 10083 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:56.184 90444 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:35:56.209 90444 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'first_report': True} sysinv 2020-05-20 16:35:56.221 10083 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'first_report': True} sysinv 2020-05-20 16:35:56.222 10083 INFO sysinv.agent.manager [-] Agent found matching ihost: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-20 16:35:56.222 10083 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-20 16:35:56.657 90444 INFO sysinv.conductor.manager [-] port 3a2c3d57-f49c-4d28-81e8-0d4172d8024b update attr: {'speed': u'1000', 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:56.935 90444 INFO sysinv.conductor.manager [-] port 908e869d-e06a-4db2-9861-d571bdc75bc3 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:57.074 90444 INFO sysinv.conductor.manager [-] port 1fad32fc-c0e3-480d-96de-5281b9d39247 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:57.211 90444 INFO sysinv.conductor.manager [-] port 8eacf1e6-edc7-4574-ab4b-12f77b656788 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:57.475 90444 INFO sysinv.conductor.manager [-] port 525c5e74-37ac-4112-be6c-fa2f81da5b40 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:57.612 90444 INFO sysinv.conductor.manager [-] port 5af31b58-1bdf-4868-944c-1ae0f4f5cc01 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:35:57.676 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ec', 'pclass': u'Unassigned class [ff00]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'ff0000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.693 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a1d2', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.710 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'-p30', 'pdevice_id': u'a1af', 'pclass': u'USB controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0330', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.729 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1b1', 'pclass': u'Signal processing controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'118000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.744 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'mei_me', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ba', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.760 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1bb', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.898 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1be', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.915 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a182', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.931 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a1', 'pclass': u'Memory controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'058000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.947 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'i801_smbus', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a3', 'pclass': u'SMBus', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0500', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.962 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a4', 'pclass': u'Serial bus controller [0c80]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c8000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.980 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'2000', 'pclass': u'VGA compatible controller', 'psdevice': u'Device 86ed', 'sriov_vfs_pci_address': u'', 'pvendor': u'ASPEED Technology, Inc.', 'pvendor_id': u'1a03', 'pclass_id': u'030000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:57.995 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'2701', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [Optane] x4 AIC (P4800X)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:58.011 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:58.029 90444 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:35:58.099 90444 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0, 1] sysinv 2020-05-20 16:35:58.099 90444 INFO sysinv.conductor.manager [-] Already in db numa_node=1 mynuma_nodes=[0, 1] sysinv 2020-05-20 16:35:58.149 90444 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:2, cores/socket=16, threads/core=2, reference:current (unchanged) sysinv 2020-05-20 16:35:58.149 90444 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 sysinv 2020-05-20 16:35:58.149 90444 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-20 16:35:58.149 90444 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 sysinv 2020-05-20 16:35:58.150 90444 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-20 16:35:58.157 90444 INFO sysinv.conductor.manager [-] update_grub_config, host uuid: (87ae4fec-0a79-4c6d-b963-35cd9d8f0262), force: (True) sysinv 2020-05-20 16:35:58.166 90444 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '4c2d9379-8aa9-4192-b533-9c2979f422bc'. sysinv 2020-05-20 16:35:58.175 90444 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 4c2d9379-8aa9-4192-b533-9c2979f422bc, applied 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-20 16:35:58.175 90444 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 64f473fa-1ab0-4079-b619-fdb925df87e9 vs. target: 4c2d9379-8aa9-4192-b533-9c2979f422bc. fmAPI.cpp(137): Connected to FM Manager. sysinv 2020-05-20 16:35:58.177 90444 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:35:58.177 90444 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=4c2d9379-8aa9-4192-b533-9c2979f422bc, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime'] sysinv 2020-05-20 16:35:58.184 90444 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:35:59.151 90444 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-20 16:35:59.151 90444 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-20 16:36:00.843 90444 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-20 16:36:00.846 90444 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-20 16:36:00.846 90444 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 4c2d9379-8aa9-4192-b533-9c2979f422bc {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} to agent sysinv 2020-05-20 16:36:02.630 90444 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': 0, 'name': u'enp1s0f0'}] sysinv 2020-05-20 16:36:02.635 10083 INFO sysinv.agent.manager [-] Affine platform interface enp1s0f0 with cpulist 0,1 sysinv 2020-05-20 16:36:02.737 10083 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp1s0f0 0,1 return 0 sysinv 2020-05-20 16:36:02.759 10083 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:aceda3e9b5e8 sysinv 2020-05-20 16:36:02.764 90444 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'config_applied': u'64f473fa-1ab0-4079-b619-fdb925df87e9', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-20 16:36:02.816 90444 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 4c2d9379-8aa9-4192-b533-9c2979f422bc, applied 64f473fa-1ab0-4079-b619-fdb925df87e9 sysinv 2020-05-20 16:36:02.816 90444 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 64f473fa-1ab0-4079-b619-fdb925df87e9 vs. target: 4c2d9379-8aa9-4192-b533-9c2979f422bc. sysinv 2020-05-20 16:36:02.821 10083 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': '64f473fa-1ab0-4079-b619-fdb925df87e9', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-20 16:36:03.721 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-36-03 patch sysinv 2020-05-20 16:36:03.722 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-20 16:36:03.740 91045 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-36-03 patch sysinv 2020-05-20 16:36:03.790 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-36-03 patch sysinv 2020-05-20 16:36:03.790 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-20 16:36:03.809 91045 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-36-03 patch sysinv 2020-05-20 16:36:05.404 10083 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:5e:00.0-nvme-1: 698 ... large disk defaults sysinv 2020-05-20 16:36:05.420 90444 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-05-20 16:36:05.420 90444 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-05-20 16:36:05.420 90444 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-05-20 16:36:05.420 90444 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-05-20 16:36:05.424 10083 INFO sysinv.agent.manager [-] Initial inventory completed host 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-20 16:36:05.441 10083 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 4c2d9379-8aa9-4192-b533-9c2979f422bc {u'classes': [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} controller sysinv 2020-05-20 16:36:05.441 10083 INFO sysinv.agent.manager [-] controller-active sysinv 2020-05-20 16:36:05.441 10083 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-05-20 16:36:11.380 10083 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::compute::grub::runtime', u'platform::compute::config::runtime']. sysinv 2020-05-20 16:36:11.381 10083 INFO sysinv.agent.manager [-] Agent config applied 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:36:11.409 90444 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:36:19.971 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-36-19 patch sysinv 2020-05-20 16:36:19.971 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-20 16:36:19.990 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_application sysinv 2020-05-20 16:36:19.993 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_controller sysinv 2020-05-20 16:36:20.279 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_worker sysinv 2020-05-20 16:36:20.799 91045 INFO sysinv.api.controllers.v1.host [-] Memory: Total=96898 MiB, Allocated=7000 MiB, 2M: 44949 pages None pages pending, 1G: 87 pages None pages pending sysinv 2020-05-20 16:36:20.827 91045 INFO sysinv.api.controllers.v1.host [-] Memory: Total=98304 MiB, Allocated=1000 MiB, 2M: 48652 pages None pages pending, 1G: 95 pages None pages pending sysinv 2020-05-20 16:36:20.839 91045 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(1): vm_mem_mib=89898, sysinv 2020-05-20 16:36:20.840 91045 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(1): {'vm_hugepages_nr_4K': 23013888} sysinv 2020-05-20 16:36:20.854 91045 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(2): vm_mem_mib=97304, sysinv 2020-05-20 16:36:20.854 91045 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(2): {'vm_hugepages_nr_4K': 24909824} sysinv 2020-05-20 16:36:21.663 91045 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query_hosts/ hdr:None payload:None sysinv 2020-05-20 16:36:22.079 91045 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'data': [{u'sw_version': u'19.12', u'hostname': u'controller-0', u'nodetype': u'controller', u'patch_failed': False, u'allow_insvc_patching': True, u'subfunctions': [u'controller', u'worker'], u'requires_reboot': False, u'installed': {}, u'state': u'idle', u'interim_state': False, u'secs_since_ack': 20, u'missing_pkgs': [], u'ip': u'192.168.204.2', u'stale_details': False, u'to_remove': [], u'patch_current': True}]} sysinv 2020-05-20 16:36:22.079 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 _semantic_mtc_check_action unlock sysinv 2020-05-20 16:36:22.079 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 action=unlock ihost_val_prenotify: {'ihost_action': 'unlock'} ihost_val: {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-20 16:36:22.079 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'unlock'} sysinv 2020-05-20 16:36:22.080 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=unlock, notify_vim=True notify_mtce=True rc=True sysinv 2020-05-20 16:36:22.080 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:36:22.080 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action unlock sysinv 2020-05-20 16:36:22.080 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_unlock_action sysinv 2020-05-20 16:36:22.080 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: unlock sysinv 2020-05-20 16:36:22.081 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:36:22.081 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-20 16:36:22.081 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-20 16:36:22.081 91045 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'unlock'} sysinv 2020-05-20 16:36:22.092 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-20 16:36:22.093 91045 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=unlock sysinv 2020-05-20 16:36:22.093 91045 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock sysinv 2020-05-20 16:36:22.093 91045 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'unlock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-20 16:36:22.093 91045 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "unlock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-20 16:36:22.097 91045 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-20 16:36:22.098 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 Perform configure_ihost. sysinv 2020-05-20 16:36:22.134 91045 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:22.139 90444 INFO sysinv.conductor.manager [-] configure_ihost controller-0 sysinv 2020-05-20 16:36:23.377 90444 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: None sysinv 2020-05-20 16:36:24.402 90444 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-20 16:36:24.402 90444 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-20 16:36:25.052 90444 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-20 16:36:25.075 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 Action unlock perform notify_mtce sysinv 2020-05-20 16:36:25.076 91045 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-20 16:36:25.076 91045 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "config_status": null, "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "online", "invprovision": "provisioned", "peer_id": null, "administrative": "locked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "target_load": "19.12", "vsc_controllers": null, "operational": "disabled", "subfunction_avail": "online", "action": "unlock", "bm_type": "none"} sysinv 2020-05-20 16:36:25.078 91045 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-20 16:36:25.093 91045 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-36-25 patch sysinv 2020-05-20 16:36:25.123 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-36-25 patch sysinv 2020-05-20 16:36:25.124 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail', 'administrative', 'availability'] sysinv 2020-05-20 16:36:25.124 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['administrative'] sysinv 2020-05-20 16:36:25.124 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-20 16:36:25.142 91045 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-36-25 patch sysinv 2020-05-20 16:36:25.166 91045 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:36:39.774 90600 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-20 16:36:39.774 90600 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-20 16:36:39.775 90600 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-20 16:36:39.775 91045 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-20 16:36:39.785 90600 INFO oslo_service.service [-] Child 91045 exited with status 0 sysinv 2020-05-20 16:36:39.786 90600 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-20 16:36:39.787 90600 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-20 16:36:39.787 90600 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-20 16:36:39.787 91046 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-20 16:36:39.790 90600 INFO oslo_service.service [-] Child 91046 exited with status 0 sysinv 2020-05-20 16:36:40.192 90444 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '55148b13-c4d6-4e1c-883a-d9c1c4276f6a'. sysinv 2020-05-20 16:36:40.204 90444 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 55148b13-c4d6-4e1c-883a-d9c1c4276f6a, applied 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:36:40.204 90444 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 4c2d9379-8aa9-4192-b533-9c2979f422bc vs. target: 55148b13-c4d6-4e1c-883a-d9c1c4276f6a. fmSocket.cpp(140): Socket Error: Failed to write to fd:(12), len:(1584), rc:(-1), error:(Broken pipe) sysinv 2020-05-20 16:36:40.205 90444 WARNING root [-] fm_python_extension: Failed to connect to FM manager sysinv 2020-05-20 16:36:40.214 90444 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=55148b13-c4d6-4e1c-883a-d9c1c4276f6a sysinv 2020-05-20 16:36:40.214 90444 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=55148b13-c4d6-4e1c-883a-d9c1c4276f6a, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'] sysinv 2020-05-20 16:36:40.221 90444 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 55148b13-c4d6-4e1c-883a-d9c1c4276f6a sysinv 2020-05-20 16:36:41.275 90444 INFO sysinv.openstack.common.service [-] Caught SIGTERM, exiting sysinv 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 104] Connection reset by peer: error: [Errno 104] Connection reset by peer 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common Traceback (most recent call last): 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 564, in ensure 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return method(*args, **kwargs) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 644, in _consume 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return connection.drain_events(**kwargs) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 464, in drain_events 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return self.blocking_read(timeout) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 469, in blocking_read 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return self.on_inbound_frame(frame) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common callback(channel, method_sig, buf, None) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 473, in on_inbound_method 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common method_sig, payload, content, 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common listener(*args) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 593, in _on_close 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common self._x_close_ok() 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 608, in _x_close_ok 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common conn.frame_writer(1, self.channel_id, sig, args, content) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common write(view[:offset]) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 301, in write 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common self._write(s) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common tail = self.send(data, flags) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return self._send_loop(self.fd.send, data, flags) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common return send_method(data, *args) 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common error: [Errno 104] Connection reset by peer 2020-05-20 16:36:44.769 10083 ERROR sysinv.openstack.common.rpc.common sysinv 2020-05-20 16:36:44.771 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:44.777 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:45.779 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:45.785 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:48.788 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:48.795 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:53.796 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:53.802 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:53.817 10083 ERROR sysinv.openstack.common.rpc.common [-] Failed to declare consumer for topic 'd9b5cde794654e998b3768ab72543038': [Errno 104] Connection reset by peer: error: [Errno 104] Connection reset by peer sysinv 2020-05-20 16:36:53.817 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:53.823 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:54.825 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:54.832 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:36:57.833 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:36:57.839 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:37:00.804 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:37:00.811 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:37:02.840 10083 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:37:02.846 10083 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:40:59.761 10075 INFO sysinv.agent.lldp.manager [-] Configured sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:40:59.763 10075 INFO sysinv.agent.lldp.manager [-] Loaded sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:40:59.763 10075 INFO sysinv.agent.lldp.manager [-] Registered sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-20 16:40:59.821 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:00.823 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:00.829 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:03.832 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:03.838 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:08.842 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:08.848 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:15.850 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:15.856 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:24.861 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:24.867 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:35.873 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:35.879 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:41:48.888 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:41:48.894 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:42:03.899 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:42:03.905 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:42:20.918 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:42:20.924 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:42:39.938 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:42:39.945 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:00.950 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:00.956 10075 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.070 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.071 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.072 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.072 90339 ERROR sysinv.common.kubernetes [-] Failed to delete custom object, Namespace kube-system: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:43:10.073 90339 WARNING sysinv.conductor.kube_app [-] Failed to clear Armada locks.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:43:10.123 90339 INFO sysinv.conductor.manager [-] sysinv-conductor start committed system={'system_mode': u'simplex', 'created_at': datetime.datetime(2020, 4, 21, 9, 57, 41, 963891, tzinfo=), 'uuid': u'1004aa49-f471-4897-97d4-8ba085382f03', 'software_version': u'19.12', 'service_project_name': u'services', 'system_type': u'All-in-one', 'name': u'3609e439-df7e-4046-bcd7-dabd81fb2f18', 'description': None, 'location': None, 'updated_at': datetime.datetime(2020, 4, 21, 9, 59, 7, 193953, tzinfo=), 'capabilities': {u'https_enabled': False, u'vswitch_type': u'none', u'region_config': False, u'sdn_enabled': False, u'shared_services': u'[]'}, 'id': 1, 'contact': None, 'security_feature': u'nopti nospectre_v2', 'services': 72, 'timezone': u'UTC', 'security_profile': u'standard', 'distributed_cloud_role': None, 'region_name': u'RegionOne'} sysinv 2020-05-20 16:43:10.148 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:10.155 90339 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-20 16:43:10.163 90493 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. sysinv 2020-05-20 16:43:10.164 90493 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". sysinv 2020-05-20 16:43:10.170 90493 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-20 16:43:10.173 90493 INFO oslo.service.wsgi [-] sysinv_api listening on 192.168.204.1:6385 sysinv 2020-05-20 16:43:10.174 90493 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-20 16:43:10.182 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.182 90493 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.183 90493 WARNING keystonemiddleware._common.config [-] The option "auth_url" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.183 90493 WARNING keystonemiddleware._common.config [-] The option "project_domain_name" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.183 90493 WARNING keystonemiddleware._common.config [-] The option "username" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.184 90493 WARNING keystonemiddleware._common.config [-] The option "project_name" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.184 90493 WARNING keystonemiddleware._common.config [-] The option "password" in conf is not known to auth_token sysinv 2020-05-20 16:43:10.186 90493 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-20 16:43:10.186 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.187 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-20 16:43:10.188 90339 ERROR sysinv.conductor.manager [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)). Failed to upgrade/downgrade tiller.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-20 16:43:10.189 90493 INFO oslo.service.wsgi [-] sysinv_api_pxe listening on 169.254.202.1:6385 sysinv 2020-05-20 16:43:10.189 90493 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-20 16:43:23.973 10075 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:23.981 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:23.990 10075 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-20 16:43:23.990 10075 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. sysinv 2020-05-20 16:43:24.742 10075 WARNING sysinv.agent.pci [-] Enabling device enp1s0f1 to query link speed sysinv 2020-05-20 16:43:24.748 10075 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp1s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:24.748 10075 WARNING sysinv.agent.pci [-] Disabling device enp1s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:25.020 90934 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. sysinv 2020-05-20 16:43:25.095 10075 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f0 sysinv 2020-05-20 16:43:25.425 10075 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f1 sysinv 2020-05-20 16:43:25.771 10075 WARNING sysinv.agent.pci [-] Enabling device enp134s0f0 to query link speed sysinv 2020-05-20 16:43:25.778 10075 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f0 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:25.778 10075 WARNING sysinv.agent.pci [-] Disabling device enp134s0f0 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:26.121 10075 WARNING sysinv.agent.pci [-] Enabling device enp134s0f1 to query link speed sysinv 2020-05-20 16:43:26.312 10075 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:26.313 10075 WARNING sysinv.agent.pci [-] Disabling device enp134s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-20 16:43:26.382 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:26.403 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:43:26.434 90339 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'first_report': True} sysinv 2020-05-20 16:43:26.446 10075 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'first_report': True} sysinv 2020-05-20 16:43:26.447 10075 INFO sysinv.agent.manager [-] Agent found matching ihost: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-20 16:43:26.447 10075 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-20 16:43:26.914 90339 INFO sysinv.conductor.manager [-] port 3a2c3d57-f49c-4d28-81e8-0d4172d8024b update attr: {'speed': u'1000', 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:43:27.205 90339 INFO sysinv.conductor.manager [-] port 908e869d-e06a-4db2-9861-d571bdc75bc3 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.220 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-43-27 patch sysinv 2020-05-20 16:43:27.220 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:43:27.354 90339 INFO sysinv.conductor.manager [-] port 1fad32fc-c0e3-480d-96de-5281b9d39247 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.509 90339 INFO sysinv.conductor.manager [-] port 8eacf1e6-edc7-4574-ab4b-12f77b656788 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.783 90339 INFO sysinv.conductor.manager [-] port 525c5e74-37ac-4112-be6c-fa2f81da5b40 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.930 90339 INFO sysinv.conductor.manager [-] port 5af31b58-1bdf-4868-944c-1ae0f4f5cc01 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-20 16:43:27.999 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ec', 'pclass': u'Unassigned class [ff00]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'ff0000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.015 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a1d2', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.030 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'-p30', 'pdevice_id': u'a1af', 'pclass': u'USB controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0330', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.047 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1b1', 'pclass': u'Signal processing controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'118000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.063 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'mei_me', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ba', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.066 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-43-28 patch sysinv 2020-05-20 16:43:28.066 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-20 16:43:28.080 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1bb', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.086 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-43-28 patch sysinv 2020-05-20 16:43:28.140 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:43:28.222 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1be', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.240 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a182', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.259 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a1', 'pclass': u'Memory controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'058000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.276 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'i801_smbus', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a3', 'pclass': u'SMBus', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0500', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.293 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a4', 'pclass': u'Serial bus controller [0c80]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c8000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.314 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'2000', 'pclass': u'VGA compatible controller', 'psdevice': u'Device 86ed', 'sriov_vfs_pci_address': u'', 'pvendor': u'ASPEED Technology, Inc.', 'pvendor_id': u'1a03', 'pclass_id': u'030000', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.332 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'2701', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [Optane] x4 AIC (P4800X)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.350 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.371 90339 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-20 16:43:28.447 90339 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0, 1] sysinv 2020-05-20 16:43:28.448 90339 INFO sysinv.conductor.manager [-] Already in db numa_node=1 mynuma_nodes=[0, 1] sysinv 2020-05-20 16:43:28.509 90339 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:2, cores/socket=16, threads/core=2, reference:current (unchanged) sysinv 2020-05-20 16:43:28.510 90339 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 sysinv 2020-05-20 16:43:28.510 90339 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-20 16:43:28.510 90339 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 sysinv 2020-05-20 16:43:28.510 90339 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-20 16:43:30.290 90339 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': 0, 'name': u'enp1s0f0'}] sysinv 2020-05-20 16:43:30.295 10075 INFO sysinv.agent.manager [-] Affine platform interface enp1s0f0 with cpulist 0,1 sysinv 2020-05-20 16:43:30.400 10075 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp1s0f0 0,1 return 0 sysinv 2020-05-20 16:43:30.423 10075 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:aceda3e9b5e8 sysinv 2020-05-20 16:43:30.427 90339 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'config_applied': u'4c2d9379-8aa9-4192-b533-9c2979f422bc', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-20 16:43:30.486 90339 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 55148b13-c4d6-4e1c-883a-d9c1c4276f6a, applied 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-20 16:43:30.487 90339 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 4c2d9379-8aa9-4192-b533-9c2979f422bc vs. target: 55148b13-c4d6-4e1c-883a-d9c1c4276f6a. fmAPI.cpp(137): Connected to FM Manager. sysinv 2020-05-20 16:43:30.493 10075 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': '4c2d9379-8aa9-4192-b533-9c2979f422bc', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-20 16:43:33.254 10075 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:5e:00.0-nvme-1: 698 ... large disk defaults sysinv 2020-05-20 16:43:33.278 90339 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-05-20 16:43:33.279 90339 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-05-20 16:43:33.279 90339 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-05-20 16:43:33.279 90339 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-05-20 16:43:33.283 10075 INFO sysinv.agent.manager [-] Initial inventory completed host 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-20 16:43:43.234 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:43:43.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:43:43.716 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:43:43.782 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-43-43 patch sysinv 2020-05-20 16:43:43.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-20 16:43:43.855 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-43-43 patch sysinv 2020-05-20 16:43:43.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:44:30.787 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-20 16:44:30.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-44-30 patch sysinv 2020-05-20 16:44:30.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:44:30.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-44-30 patch sysinv 2020-05-20 16:44:30.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-20 16:44:31.151 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-44-31 patch sysinv 2020-05-20 16:45:10.616 90339 INFO sysinv.conductor.manager [-] Audit clearing vim_progress_status=services-disable-failed.. sysinv 2020-05-20 16:46:21.043 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-46-21 patch sysinv 2020-05-20 16:46:21.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-20 16:46:21.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-enabled sysinv 2020-05-20 16:46:21.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-enabled ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-20 16:46:21.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-enabled, notify_vim=False notify_mtce=False rc=True sysinv 2020-05-20 16:46:21.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-enabled sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 received services-enabled task= vim_progress_status= sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-enabled sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-20 16:46:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-20 16:46:21.046 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-enabled'} sysinv 2020-05-20 16:46:21.107 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 notify_availability=services-enabled sysinv 2020-05-20 16:46:21.142 90934 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 16:46:21.146 90339 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'availability': u'services-enabled'} sysinv 2020-05-20 16:46:21.190 90339 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-20 16:46:21.195 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-20-16-46-21 patch sysinv 2020-05-20 16:54:31.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-54-30 patch sysinv 2020-05-20 16:54:31.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 16:59:31.046 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-16-59-31 patch sysinv 2020-05-20 16:59:31.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:04:31.082 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-04-31 patch sysinv 2020-05-20 17:04:31.082 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:09:31.106 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-09-31 patch sysinv 2020-05-20 17:09:31.106 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:14:31.237 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-14-31 patch sysinv 2020-05-20 17:14:31.237 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:19:31.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-19-31 patch sysinv 2020-05-20 17:19:31.254 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:24:31.183 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-24-31 patch sysinv 2020-05-20 17:24:31.183 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:29:31.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-29-31 patch sysinv 2020-05-20 17:29:31.310 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:34:31.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-34-31 patch sysinv 2020-05-20 17:34:31.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:39:31.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-39-31 patch sysinv 2020-05-20 17:39:31.363 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:43:10.229 90339 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-20 17:43:10.239 90339 INFO sysinv.conductor.manager [-] Running tiller image: registry.local:9001/gcr.io/kubernetes-helm/tiller:v2.13.1 sysinv 2020-05-20 17:43:10.239 90339 INFO sysinv.conductor.manager [-] Requested tiller version: v2.13.1 sysinv 2020-05-20 17:43:10.239 90339 INFO sysinv.conductor.manager [-] _upgrade_downgrade_kube_networking executing playbook: /usr/share/ansible/stx-ansible/playbooks/upgrade-k8s-networking.yml sysinv 2020-05-20 17:44:27.326 10075 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 17:44:27.336 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:44:31.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-44-31 patch sysinv 2020-05-20 17:44:31.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 17:45:27.398 10075 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 17:45:27.400 10075 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.141628 sec sysinv 2020-05-20 17:45:27.412 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:46:27.478 10075 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 17:46:27.493 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task [-] Error during AgentManager._agent_audit: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "": Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1122, in _agent_audit 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task force_updates=None) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task return f(*args, **kwargs) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 1213, in agent_audit 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task self._update_ttys_dcd_status(icontext, self._ihost_uuid) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/agent/manager.py", line 280, in _update_ttys_dcd_status 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task ttys_dcd = rpcapi.get_host_ttys_dcd(context, host_id) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1171, in get_host_ttys_dcd 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task ihost_id=ihost_id)) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/proxy.py", line 126, in call 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task exc.info, real_topic, msg.get('method')) 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "get_host_ttys_dcd" info: "" 2020-05-20 17:47:27.560 10075 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-20 17:47:27.561 10075 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 60.16004 sec sysinv 2020-05-20 17:47:27.570 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:48:27.632 10075 INFO sysinv.agent.manager [-] ilvg_get_nova_ilvg_by_ihost() Timeout. sysinv 2020-05-20 17:48:27.642 10075 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:49:03.023 90339 INFO sysinv.conductor.manager [-] ansible-playbook: PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Create the upgrade overrides file] *************************************** changed: [localhost] TASK [Read the overrides variables] ******************************************** ok: [localhost] TASK [Set networking facts] **************************************************** ok: [localhost] TASK [Set images facts] ******************************************************** ok: [localhost] TASK [common/push-docker-images : Get docker registries if upgrade kubernetes or k8s networking] *** included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'k8s_registry', u'value': {u'url': u'k8s.gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'gcr_registry', u'value': {u'url': u'gcr.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'quay_registry', u'value': {u'url': u'quay.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'docker_registry', u'value': {u'url': u'docker.io'}}) included: /usr/share/ansible/stx-ansible/playbooks/roles/common/push-docker-images/tasks/get_docker_registry.yml for localhost => (item={u'name': u'elastic_registry', u'value': {u'url': u'docker.elastic.co'}}) TASK [common/push-docker-images : Query the k8s_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate k8s_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate k8s_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the gcr_registry] ********************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate gcr_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the gcr_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate gcr_registry secret] **************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the quay_registry] ********************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate quay_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the quay_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate quay_registry secret] *************** TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the docker_registry] ******************* changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate docker_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the docker_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate docker_registry secret] ************* TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Query the elastic_registry] ****************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : Validate elastic_registry information if it exists] *** TASK [common/push-docker-images : Set secure to bool] ************************** TASK [common/push-docker-images : Get the elastic_registry barbican secret if it's authenticated] *** TASK [common/push-docker-images : Validate elastic_registry secret] ************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get registry credentials if registry type is AWS ECR] *** TASK [common/push-docker-images : Get kubernetes version from template] ******** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of kubernetes images] *********** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Get the list of platform images] ************* ok: [localhost] TASK [common/push-docker-images : Set download images list] ******************** TASK [common/push-docker-images : Set download images list to k8s network images if upgrading k8s networking] *** ok: [localhost] TASK [common/push-docker-images : Set download images list to kubernetes images if upgrading kubernetes] *** TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "download_images_list": [ "quay.io/calico/cni:v3.6.2", "quay.io/calico/node:v3.6.2", "quay.io/calico/kube-controllers:v3.6.2", "docker.io/starlingx/multus:v3.2.16", "docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest" ] } TASK [common/push-docker-images : Set registries information] ****************** ok: [localhost] => (item={u'replaced_url': u'k8s.gcr.io', u'default_url': u'k8s.gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'gcr.io', u'default_url': u'gcr.io'}) ok: [localhost] => (item={u'replaced_url': u'quay.io', u'default_url': u'quay.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.io', u'default_url': u'docker.io'}) ok: [localhost] => (item={u'replaced_url': u'docker.elastic.co', u'default_url': u'docker.elastic.co'}) TASK [common/push-docker-images : Log in k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Get local registry credentials] ************** changed: [localhost] TASK [common/push-docker-images : set_fact] ************************************ ok: [localhost] TASK [common/push-docker-images : Log in to local registry] ******************** ok: [localhost] TASK [common/push-docker-images : Download images and push to local registry] *** changed: [localhost] TASK [common/push-docker-images : debug] *************************************** ok: [localhost] => { "msg": [ "Image download succeeded: quay.io/calico/cni:v3.6.2", "Image push succeeded: registry.local:9001/quay.io/calico/cni:v3.6.2", "Image download succeeded: quay.io/calico/node:v3.6.2", "Image push succeeded: registry.local:9001/quay.io/calico/node:v3.6.2", "Image download succeeded: quay.io/calico/kube-controllers:v3.6.2", "Image push succeeded: registry.local:9001/quay.io/calico/kube-controllers:v3.6.2", "Image download succeeded: docker.io/starlingx/multus:v3.2.16", "Image push succeeded: registry.local:9001/docker.io/starlingx/multus:v3.2.16", "Image download succeeded: docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "Image push succeeded: registry.local:9001/docker.io/starlingx/k8s-cni-sriov:master-centos-stable-latest", "Image download succeeded: docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest", "Image push succeeded: registry.local:9001/docker.io/starlingx/k8s-plugins-sriov-network-device:master-centos-stable-latest", "All images downloaded and pushed to the local registry in 312.239131927 seconds" ] } TASK [common/push-docker-images : Log out of k8s, gcr, quay, docker registries if credentials exist] *** TASK [common/push-docker-images : Log out of local registry] ******************* ok: [localhost] TASK [Create Calico config file] *********************************************** ok: [localhost] TASK [Update Calico Networking] ************************************************ changed: [localhost] TASK [Create Multus config file] *********************************************** ok: [localhost] TASK [Update Multus Networking] ************************************************ changed: [localhost] TASK [Create SRIOV Networking config file] ************************************* ok: [localhost] TASK [Update SRIOV Networking] ************************************************* changed: [localhost] TASK [Create SRIOV device plugin config file] ********************************** ok: [localhost] TASK [Update SRIOV device plugin] ********************************************** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=48 changed=14 unreachable=0 failed=0 . sysinv 2020-05-20 17:49:03.134 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:49:03.207 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:49:03.211 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:49:03.214 90339 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-20 17:49:05.989 10075 WARNING sysinv.openstack.common.loopingcall [-] task run outlasted interval by 38.427663 sec sysinv 2020-05-20 17:49:31.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-49-31 patch sysinv 2020-05-20 17:49:31.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:54:31.507 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-54-31 patch sysinv 2020-05-20 17:54:31.508 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 17:59:31.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-17-59-31 patch sysinv 2020-05-20 17:59:31.568 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:04:31.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-04-31 patch sysinv 2020-05-20 18:04:31.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:09:31.663 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-09-31 patch sysinv 2020-05-20 18:09:31.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:14:31.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-14-31 patch sysinv 2020-05-20 18:14:31.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:19:31.577 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-19-31 patch sysinv 2020-05-20 18:19:31.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:24:31.545 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-24-31 patch sysinv 2020-05-20 18:24:31.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:29:31.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-29-31 patch sysinv 2020-05-20 18:29:31.633 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:34:31.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-34-31 patch sysinv 2020-05-20 18:34:31.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:39:31.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-39-31 patch sysinv 2020-05-20 18:39:31.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:44:31.677 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-44-31 patch sysinv 2020-05-20 18:44:31.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:49:31.832 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-49-31 patch sysinv 2020-05-20 18:49:31.833 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:54:31.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-54-31 patch sysinv 2020-05-20 18:54:31.839 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 18:59:31.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-18-59-31 patch sysinv 2020-05-20 18:59:31.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:04:31.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-04-31 patch sysinv 2020-05-20 19:04:31.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:09:31.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-09-31 patch sysinv 2020-05-20 19:09:31.956 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:14:31.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-14-31 patch sysinv 2020-05-20 19:14:31.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:19:31.963 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-19-31 patch sysinv 2020-05-20 19:19:31.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:24:31.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-24-31 patch sysinv 2020-05-20 19:24:31.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:29:32.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-29-32 patch sysinv 2020-05-20 19:29:32.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:34:31.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-34-31 patch sysinv 2020-05-20 19:34:31.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:39:32.013 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-39-31 patch sysinv 2020-05-20 19:39:32.013 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:44:32.111 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-44-32 patch sysinv 2020-05-20 19:44:32.111 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:49:32.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-49-32 patch sysinv 2020-05-20 19:49:32.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:54:32.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-54-32 patch sysinv 2020-05-20 19:54:32.310 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 19:59:32.193 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-19-59-32 patch sysinv 2020-05-20 19:59:32.193 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:04:32.223 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-04-32 patch sysinv 2020-05-20 20:04:32.223 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:09:32.351 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-09-32 patch sysinv 2020-05-20 20:09:32.356 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:14:32.373 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-14-32 patch sysinv 2020-05-20 20:14:32.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:19:32.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-19-32 patch sysinv 2020-05-20 20:19:32.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:24:32.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-24-32 patch sysinv 2020-05-20 20:24:32.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:29:32.390 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-29-32 patch sysinv 2020-05-20 20:29:32.391 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:34:32.412 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-34-32 patch sysinv 2020-05-20 20:34:32.413 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:39:32.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-39-32 patch sysinv 2020-05-20 20:39:32.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:44:32.446 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-44-32 patch sysinv 2020-05-20 20:44:32.446 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:49:32.505 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-49-32 patch sysinv 2020-05-20 20:49:32.506 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:54:32.541 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-54-32 patch sysinv 2020-05-20 20:54:32.542 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 20:59:32.548 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-20-59-32 patch sysinv 2020-05-20 20:59:32.548 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:04:32.583 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-04-32 patch sysinv 2020-05-20 21:04:32.583 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:09:32.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-09-32 patch sysinv 2020-05-20 21:09:32.710 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:14:32.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-14-32 patch sysinv 2020-05-20 21:14:32.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:19:32.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-19-32 patch sysinv 2020-05-20 21:19:32.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:24:32.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-24-32 patch sysinv 2020-05-20 21:24:32.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:29:32.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-29-32 patch sysinv 2020-05-20 21:29:32.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:34:32.796 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-34-32 patch sysinv 2020-05-20 21:34:32.796 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:39:32.917 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-39-32 patch sysinv 2020-05-20 21:39:32.918 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:44:32.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-44-32 patch sysinv 2020-05-20 21:44:32.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:49:32.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-49-32 patch sysinv 2020-05-20 21:49:32.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:54:32.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-54-32 patch sysinv 2020-05-20 21:54:32.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 21:59:32.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-21-59-32 patch sysinv 2020-05-20 21:59:32.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:04:32.882 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-04-32 patch sysinv 2020-05-20 22:04:32.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:09:33.068 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-09-33 patch sysinv 2020-05-20 22:09:33.069 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:14:32.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-14-32 patch sysinv 2020-05-20 22:14:32.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:19:32.975 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-19-32 patch sysinv 2020-05-20 22:19:32.976 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:24:33.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-24-33 patch sysinv 2020-05-20 22:24:33.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:29:33.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-29-33 patch sysinv 2020-05-20 22:29:33.073 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:34:33.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-34-33 patch sysinv 2020-05-20 22:34:33.157 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:39:33.166 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-39-33 patch sysinv 2020-05-20 22:39:33.167 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:44:33.563 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-44-33 patch sysinv 2020-05-20 22:44:33.563 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:49:33.192 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-49-33 patch sysinv 2020-05-20 22:49:33.193 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:54:33.275 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-54-33 patch sysinv 2020-05-20 22:54:33.276 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 22:59:33.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-22-59-33 patch sysinv 2020-05-20 22:59:33.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:04:33.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-04-33 patch sysinv 2020-05-20 23:04:33.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:09:33.354 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-09-33 patch sysinv 2020-05-20 23:09:33.356 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:14:33.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-14-33 patch sysinv 2020-05-20 23:14:33.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:19:33.438 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-19-33 patch sysinv 2020-05-20 23:19:33.439 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:24:33.406 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-24-33 patch sysinv 2020-05-20 23:24:33.406 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:29:33.586 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-29-33 patch sysinv 2020-05-20 23:29:33.586 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:34:33.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-34-33 patch sysinv 2020-05-20 23:34:33.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:39:33.511 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-39-33 patch sysinv 2020-05-20 23:39:33.512 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:44:33.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-44-33 patch sysinv 2020-05-20 23:44:33.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:49:33.562 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-49-33 patch sysinv 2020-05-20 23:49:33.562 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:54:33.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-54-33 patch sysinv 2020-05-20 23:54:33.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-20 23:59:33.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-20-23-59-33 patch sysinv 2020-05-20 23:59:33.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:04:33.703 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-04-33 patch sysinv 2020-05-21 00:04:33.703 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:09:33.676 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-09-33 patch sysinv 2020-05-21 00:09:33.677 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:14:33.794 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-14-33 patch sysinv 2020-05-21 00:14:33.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:19:33.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-19-33 patch sysinv 2020-05-21 00:19:33.784 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:24:33.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-24-33 patch sysinv 2020-05-21 00:24:33.885 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:29:33.860 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-29-33 patch sysinv 2020-05-21 00:29:33.861 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:34:34.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-34-33 patch sysinv 2020-05-21 00:34:34.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:39:33.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-39-33 patch sysinv 2020-05-21 00:39:33.912 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:44:34.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-44-34 patch sysinv 2020-05-21 00:44:34.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:49:33.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-49-33 patch sysinv 2020-05-21 00:49:33.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:54:33.907 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-54-33 patch sysinv 2020-05-21 00:54:33.907 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 00:59:33.937 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-00-59-33 patch sysinv 2020-05-21 00:59:33.938 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:04:33.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-04-33 patch sysinv 2020-05-21 01:04:33.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:09:34.150 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-09-34 patch sysinv 2020-05-21 01:09:34.150 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:14:34.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-14-33 patch sysinv 2020-05-21 01:14:34.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:19:34.051 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-19-34 patch sysinv 2020-05-21 01:19:34.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:24:34.087 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-24-34 patch sysinv 2020-05-21 01:24:34.087 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:29:34.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-29-34 patch sysinv 2020-05-21 01:29:34.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:34:34.287 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-34-34 patch sysinv 2020-05-21 01:34:34.288 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:39:34.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-39-34 patch sysinv 2020-05-21 01:39:34.235 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:44:34.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-44-34 patch sysinv 2020-05-21 01:44:34.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:49:34.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-49-34 patch sysinv 2020-05-21 01:49:34.298 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:54:34.407 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-54-34 patch sysinv 2020-05-21 01:54:34.408 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 01:59:34.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-01-59-34 patch sysinv 2020-05-21 01:59:34.390 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:04:34.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-04-34 patch sysinv 2020-05-21 02:04:34.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:09:34.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-09-34 patch sysinv 2020-05-21 02:09:34.405 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:14:34.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-14-34 patch sysinv 2020-05-21 02:14:34.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:19:34.583 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-19-34 patch sysinv 2020-05-21 02:19:34.584 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:24:34.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-24-34 patch sysinv 2020-05-21 02:24:34.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:29:34.524 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-29-34 patch sysinv 2020-05-21 02:29:34.524 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:34:34.791 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-34-34 patch sysinv 2020-05-21 02:34:34.791 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:39:34.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-39-34 patch sysinv 2020-05-21 02:39:34.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:44:34.655 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-44-34 patch sysinv 2020-05-21 02:44:34.656 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:49:34.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-49-34 patch sysinv 2020-05-21 02:49:34.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:54:34.750 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-54-34 patch sysinv 2020-05-21 02:54:34.750 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 02:59:34.757 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-02-59-34 patch sysinv 2020-05-21 02:59:34.757 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:04:34.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-04-34 patch sysinv 2020-05-21 03:04:34.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:09:34.813 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-09-34 patch sysinv 2020-05-21 03:09:34.813 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:14:34.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-14-34 patch sysinv 2020-05-21 03:14:34.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:19:34.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-19-34 patch sysinv 2020-05-21 03:19:34.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:24:34.929 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-24-34 patch sysinv 2020-05-21 03:24:34.930 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:29:34.919 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-29-34 patch sysinv 2020-05-21 03:29:34.920 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:34:35.106 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-34-35 patch sysinv 2020-05-21 03:34:35.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:39:34.902 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-39-34 patch sysinv 2020-05-21 03:39:34.904 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:44:35.376 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-44-35 patch sysinv 2020-05-21 03:44:35.376 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:49:34.966 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-49-34 patch sysinv 2020-05-21 03:49:34.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:54:34.998 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-54-34 patch sysinv 2020-05-21 03:54:34.998 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 03:59:35.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-03-59-34 patch sysinv 2020-05-21 03:59:35.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:04:35.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-04-35 patch sysinv 2020-05-21 04:04:35.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:09:35.204 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-09-35 patch sysinv 2020-05-21 04:09:35.205 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:14:35.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-14-35 patch sysinv 2020-05-21 04:14:35.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:19:35.229 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-19-35 patch sysinv 2020-05-21 04:19:35.230 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:24:35.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-24-35 patch sysinv 2020-05-21 04:24:35.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:29:35.307 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-29-35 patch sysinv 2020-05-21 04:29:35.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:34:35.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-34-35 patch sysinv 2020-05-21 04:34:35.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:39:35.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-39-35 patch sysinv 2020-05-21 04:39:35.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:44:35.716 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-44-35 patch sysinv 2020-05-21 04:44:35.717 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:49:35.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-49-35 patch sysinv 2020-05-21 04:49:35.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:54:35.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-54-35 patch sysinv 2020-05-21 04:54:35.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 04:59:35.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-04-59-35 patch sysinv 2020-05-21 04:59:35.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:04:35.466 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-04-35 patch sysinv 2020-05-21 05:04:35.466 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:09:35.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-09-35 patch sysinv 2020-05-21 05:09:35.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:14:35.554 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-14-35 patch sysinv 2020-05-21 05:14:35.555 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:19:35.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-19-35 patch sysinv 2020-05-21 05:19:35.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:24:35.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-24-35 patch sysinv 2020-05-21 05:24:35.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:29:35.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-29-35 patch sysinv 2020-05-21 05:29:35.731 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:34:35.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-34-35 patch sysinv 2020-05-21 05:34:35.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:39:35.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-39-35 patch sysinv 2020-05-21 05:39:35.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:44:35.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-44-35 patch sysinv 2020-05-21 05:44:35.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:49:35.724 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-49-35 patch sysinv 2020-05-21 05:49:35.724 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:54:35.820 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-54-35 patch sysinv 2020-05-21 05:54:35.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 05:59:35.797 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-05-59-35 patch sysinv 2020-05-21 05:59:35.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:04:35.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-04-35 patch sysinv 2020-05-21 06:04:35.951 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:09:35.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-09-35 patch sysinv 2020-05-21 06:09:35.864 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:14:35.897 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-14-35 patch sysinv 2020-05-21 06:14:35.897 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:19:35.965 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-19-35 patch sysinv 2020-05-21 06:19:35.966 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:24:35.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-24-35 patch sysinv 2020-05-21 06:24:35.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:29:36.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-29-35 patch sysinv 2020-05-21 06:29:36.025 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:34:36.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-34-36 patch sysinv 2020-05-21 06:34:36.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:39:35.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-39-35 patch sysinv 2020-05-21 06:39:35.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:44:36.070 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-44-36 patch sysinv 2020-05-21 06:44:36.071 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:49:36.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-49-36 patch sysinv 2020-05-21 06:49:36.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:54:36.223 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-54-36 patch sysinv 2020-05-21 06:54:36.224 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 06:59:36.213 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-06-59-36 patch sysinv 2020-05-21 06:59:36.214 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:04:36.137 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-04-36 patch sysinv 2020-05-21 07:04:36.138 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:09:36.382 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-09-36 patch sysinv 2020-05-21 07:09:36.383 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:14:36.327 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-14-36 patch sysinv 2020-05-21 07:14:36.327 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:19:36.292 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-19-36 patch sysinv 2020-05-21 07:19:36.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:24:36.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-24-36 patch sysinv 2020-05-21 07:24:36.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:29:36.359 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-29-36 patch sysinv 2020-05-21 07:29:36.360 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:34:36.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-34-36 patch sysinv 2020-05-21 07:34:36.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:39:36.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-39-36 patch sysinv 2020-05-21 07:39:36.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:44:36.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-44-36 patch sysinv 2020-05-21 07:44:36.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:49:36.466 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-49-36 patch sysinv 2020-05-21 07:49:36.466 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:54:36.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-54-36 patch sysinv 2020-05-21 07:54:36.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 07:59:36.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-07-59-36 patch sysinv 2020-05-21 07:59:36.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:04:36.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-04-36 patch sysinv 2020-05-21 08:04:36.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:09:36.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-09-36 patch sysinv 2020-05-21 08:09:36.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:14:36.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-14-36 patch sysinv 2020-05-21 08:14:36.628 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:19:36.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-19-36 patch sysinv 2020-05-21 08:19:36.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:24:36.655 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-24-36 patch sysinv 2020-05-21 08:24:36.655 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:29:36.766 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-29-36 patch sysinv 2020-05-21 08:29:36.766 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:34:36.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-34-36 patch sysinv 2020-05-21 08:34:36.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:39:36.870 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-39-36 patch sysinv 2020-05-21 08:39:36.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:44:37.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-44-37 patch sysinv 2020-05-21 08:44:37.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:49:36.776 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-49-36 patch sysinv 2020-05-21 08:49:36.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:54:36.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-54-36 patch sysinv 2020-05-21 08:54:36.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 08:59:36.907 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-08-59-36 patch sysinv 2020-05-21 08:59:36.907 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:04:36.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-04-36 patch sysinv 2020-05-21 09:04:36.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:09:36.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-09-36 patch sysinv 2020-05-21 09:09:36.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:14:37.059 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-14-37 patch sysinv 2020-05-21 09:14:37.059 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:19:36.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-19-36 patch sysinv 2020-05-21 09:19:36.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:24:37.250 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-24-37 patch sysinv 2020-05-21 09:24:37.251 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:29:37.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-29-36 patch sysinv 2020-05-21 09:29:37.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:34:37.053 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-34-37 patch sysinv 2020-05-21 09:34:37.054 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:39:37.079 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-39-37 patch sysinv 2020-05-21 09:39:37.080 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:44:37.488 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-44-37 patch sysinv 2020-05-21 09:44:37.489 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:49:37.237 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-49-37 patch sysinv 2020-05-21 09:49:37.237 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:54:37.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-54-37 patch sysinv 2020-05-21 09:54:37.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 09:59:37.300 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-09-59-37 patch sysinv 2020-05-21 09:59:37.300 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:04:37.228 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-04-37 patch sysinv 2020-05-21 10:04:37.229 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:09:37.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-09-37 patch sysinv 2020-05-21 10:09:37.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:14:37.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-14-37 patch sysinv 2020-05-21 10:14:37.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:19:37.493 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-19-37 patch sysinv 2020-05-21 10:19:37.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:24:37.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-24-37 patch sysinv 2020-05-21 10:24:37.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:29:37.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-29-37 patch sysinv 2020-05-21 10:29:37.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:34:37.500 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-34-37 patch sysinv 2020-05-21 10:34:37.501 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:39:37.638 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-39-37 patch sysinv 2020-05-21 10:39:37.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:44:37.547 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-44-37 patch sysinv 2020-05-21 10:44:37.547 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:49:37.778 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-49-37 patch sysinv 2020-05-21 10:49:37.778 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:54:37.644 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-54-37 patch sysinv 2020-05-21 10:54:37.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 10:59:37.726 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-10-59-37 patch sysinv 2020-05-21 10:59:37.726 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:04:37.696 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-04-37 patch sysinv 2020-05-21 11:04:37.697 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:09:37.805 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-09-37 patch sysinv 2020-05-21 11:09:37.805 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:14:37.789 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-14-37 patch sysinv 2020-05-21 11:14:37.789 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:19:37.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-19-37 patch sysinv 2020-05-21 11:19:37.759 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:24:37.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-24-37 patch sysinv 2020-05-21 11:24:37.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:29:37.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-29-37 patch sysinv 2020-05-21 11:29:37.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:34:37.898 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-34-37 patch sysinv 2020-05-21 11:34:37.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:39:37.843 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-39-37 patch sysinv 2020-05-21 11:39:37.843 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:44:38.324 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-44-38 patch sysinv 2020-05-21 11:44:38.324 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:49:37.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-49-37 patch sysinv 2020-05-21 11:49:37.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:54:37.897 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-54-37 patch sysinv 2020-05-21 11:54:37.897 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 11:59:37.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-11-59-37 patch sysinv 2020-05-21 11:59:37.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:04:38.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-04-37 patch sysinv 2020-05-21 12:04:38.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:09:38.084 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-09-38 patch sysinv 2020-05-21 12:09:38.084 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:14:38.046 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-14-38 patch sysinv 2020-05-21 12:14:38.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:19:38.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-19-38 patch sysinv 2020-05-21 12:19:38.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:24:38.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-24-38 patch sysinv 2020-05-21 12:24:38.132 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:29:38.162 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-29-38 patch sysinv 2020-05-21 12:29:38.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:34:38.257 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-34-38 patch sysinv 2020-05-21 12:34:38.258 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:39:38.304 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-39-38 patch sysinv 2020-05-21 12:39:38.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:44:38.212 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-44-38 patch sysinv 2020-05-21 12:44:38.213 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:49:38.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-49-38 patch sysinv 2020-05-21 12:49:38.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:54:38.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-54-38 patch sysinv 2020-05-21 12:54:38.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 12:59:38.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-12-59-38 patch sysinv 2020-05-21 12:59:38.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:04:38.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-04-38 patch sysinv 2020-05-21 13:04:38.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:09:38.540 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-09-38 patch sysinv 2020-05-21 13:09:38.541 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:14:38.444 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-14-38 patch sysinv 2020-05-21 13:14:38.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:19:38.557 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-19-38 patch sysinv 2020-05-21 13:19:38.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:24:38.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-24-38 patch sysinv 2020-05-21 13:24:38.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:29:38.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-29-38 patch sysinv 2020-05-21 13:29:38.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:34:38.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-34-38 patch sysinv 2020-05-21 13:34:38.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:39:38.654 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-39-38 patch sysinv 2020-05-21 13:39:38.655 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:44:38.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-44-38 patch sysinv 2020-05-21 13:44:38.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:49:38.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-49-38 patch sysinv 2020-05-21 13:49:38.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:54:38.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-54-38 patch sysinv 2020-05-21 13:54:38.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 13:59:38.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-13-59-38 patch sysinv 2020-05-21 13:59:38.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:04:38.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-04-38 patch sysinv 2020-05-21 14:04:38.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:09:38.824 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-09-38 patch sysinv 2020-05-21 14:09:38.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:14:38.864 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-14-38 patch sysinv 2020-05-21 14:14:38.864 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:19:38.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-19-38 patch sysinv 2020-05-21 14:19:38.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:24:38.898 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-24-38 patch sysinv 2020-05-21 14:24:38.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:29:38.963 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-29-38 patch sysinv 2020-05-21 14:29:38.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:34:39.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-34-38 patch sysinv 2020-05-21 14:34:39.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:39:39.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-39-38 patch sysinv 2020-05-21 14:39:39.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:44:39.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-44-39 patch sysinv 2020-05-21 14:44:39.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:49:38.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-49-38 patch sysinv 2020-05-21 14:49:38.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:54:39.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-54-38 patch sysinv 2020-05-21 14:54:39.004 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 14:59:39.021 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-14-59-39 patch sysinv 2020-05-21 14:59:39.021 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:04:39.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-04-39 patch sysinv 2020-05-21 15:04:39.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:09:39.187 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-09-39 patch sysinv 2020-05-21 15:09:39.188 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:14:39.255 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-14-39 patch sysinv 2020-05-21 15:14:39.255 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:19:39.262 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-19-39 patch sysinv 2020-05-21 15:19:39.263 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:24:39.289 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-24-39 patch sysinv 2020-05-21 15:24:39.289 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:29:39.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-29-39 patch sysinv 2020-05-21 15:29:39.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:34:39.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-34-39 patch sysinv 2020-05-21 15:34:39.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:39:39.353 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-39-39 patch sysinv 2020-05-21 15:39:39.354 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:44:39.387 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-44-39 patch sysinv 2020-05-21 15:44:39.387 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:49:39.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-49-39 patch sysinv 2020-05-21 15:49:39.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:54:39.539 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-54-39 patch sysinv 2020-05-21 15:54:39.539 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 15:59:39.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-15-59-39 patch sysinv 2020-05-21 15:59:39.495 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:04:39.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-04-39 patch sysinv 2020-05-21 16:04:39.418 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:09:39.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-09-39 patch sysinv 2020-05-21 16:09:39.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:14:39.545 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-14-39 patch sysinv 2020-05-21 16:14:39.545 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:19:39.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-19-39 patch sysinv 2020-05-21 16:19:39.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:24:39.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-24-39 patch sysinv 2020-05-21 16:24:39.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:29:39.694 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-29-39 patch sysinv 2020-05-21 16:29:39.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:34:39.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-34-39 patch sysinv 2020-05-21 16:34:39.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:39:39.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-39-39 patch sysinv 2020-05-21 16:39:39.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:44:40.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-44-40 patch sysinv 2020-05-21 16:44:40.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:49:39.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-49-39 patch sysinv 2020-05-21 16:49:39.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:54:39.894 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-54-39 patch sysinv 2020-05-21 16:54:39.895 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 16:59:39.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-16-59-39 patch sysinv 2020-05-21 16:59:39.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:04:39.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-04-39 patch sysinv 2020-05-21 17:04:39.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:09:39.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-09-39 patch sysinv 2020-05-21 17:09:39.909 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:14:39.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-14-39 patch sysinv 2020-05-21 17:14:39.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:19:40.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-19-39 patch sysinv 2020-05-21 17:19:40.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:24:39.969 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-24-39 patch sysinv 2020-05-21 17:24:39.969 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:29:40.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-29-39 patch sysinv 2020-05-21 17:29:40.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:34:40.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-34-40 patch sysinv 2020-05-21 17:34:40.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:39:39.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-39-39 patch sysinv 2020-05-21 17:39:39.997 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:44:40.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-44-40 patch sysinv 2020-05-21 17:44:40.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:49:40.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-49-40 patch sysinv 2020-05-21 17:49:40.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:54:40.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-54-40 patch sysinv 2020-05-21 17:54:40.222 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 17:59:40.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-17-59-40 patch sysinv 2020-05-21 17:59:40.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:04:40.242 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-04-40 patch sysinv 2020-05-21 18:04:40.243 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:09:40.215 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-09-40 patch sysinv 2020-05-21 18:09:40.216 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:14:40.303 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-14-40 patch sysinv 2020-05-21 18:14:40.303 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:19:40.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-19-40 patch sysinv 2020-05-21 18:19:40.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:24:40.379 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-24-40 patch sysinv 2020-05-21 18:24:40.380 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:29:40.344 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-29-40 patch sysinv 2020-05-21 18:29:40.345 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:34:40.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-34-40 patch sysinv 2020-05-21 18:34:40.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:39:40.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-39-40 patch sysinv 2020-05-21 18:39:40.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:44:40.797 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-44-40 patch sysinv 2020-05-21 18:44:40.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:49:40.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-49-40 patch sysinv 2020-05-21 18:49:40.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:54:40.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-54-40 patch sysinv 2020-05-21 18:54:40.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 18:59:40.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-18-59-40 patch sysinv 2020-05-21 18:59:40.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:04:40.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-04-40 patch sysinv 2020-05-21 19:04:40.503 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:09:40.702 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-09-40 patch sysinv 2020-05-21 19:09:40.704 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:14:40.672 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-14-40 patch sysinv 2020-05-21 19:14:40.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:19:40.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-19-40 patch sysinv 2020-05-21 19:19:40.721 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:24:40.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-24-40 patch sysinv 2020-05-21 19:24:40.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:29:40.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-29-40 patch sysinv 2020-05-21 19:29:40.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:34:40.740 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-34-40 patch sysinv 2020-05-21 19:34:40.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:39:41.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-39-40 patch sysinv 2020-05-21 19:39:41.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:44:40.800 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-44-40 patch sysinv 2020-05-21 19:44:40.800 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:49:40.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-49-40 patch sysinv 2020-05-21 19:49:40.815 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:54:40.927 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-54-40 patch sysinv 2020-05-21 19:54:40.927 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 19:59:40.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-19-59-40 patch sysinv 2020-05-21 19:59:40.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:04:40.971 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-04-40 patch sysinv 2020-05-21 20:04:40.972 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:09:41.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-09-40 patch sysinv 2020-05-21 20:09:41.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:14:41.065 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-14-41 patch sysinv 2020-05-21 20:14:41.066 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:19:40.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-19-40 patch sysinv 2020-05-21 20:19:40.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:24:41.140 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-24-41 patch sysinv 2020-05-21 20:24:41.140 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:29:41.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-29-40 patch sysinv 2020-05-21 20:29:41.015 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:34:41.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-34-41 patch sysinv 2020-05-21 20:34:41.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:39:41.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-39-41 patch sysinv 2020-05-21 20:39:41.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:44:41.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-44-41 patch sysinv 2020-05-21 20:44:41.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:49:41.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-49-41 patch sysinv 2020-05-21 20:49:41.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:54:41.241 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-54-41 patch sysinv 2020-05-21 20:54:41.241 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 20:59:41.311 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-20-59-41 patch sysinv 2020-05-21 20:59:41.311 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:04:41.303 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-04-41 patch sysinv 2020-05-21 21:04:41.304 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:09:41.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-09-41 patch sysinv 2020-05-21 21:09:41.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:14:41.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-14-41 patch sysinv 2020-05-21 21:14:41.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:19:41.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-19-41 patch sysinv 2020-05-21 21:19:41.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:24:41.447 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-24-41 patch sysinv 2020-05-21 21:24:41.447 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:29:41.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-29-41 patch sysinv 2020-05-21 21:29:41.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:34:41.577 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-34-41 patch sysinv 2020-05-21 21:34:41.577 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:39:41.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-39-41 patch sysinv 2020-05-21 21:39:41.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:44:41.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-44-41 patch sysinv 2020-05-21 21:44:41.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:49:41.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-49-41 patch sysinv 2020-05-21 21:49:41.589 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:54:41.610 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-54-41 patch sysinv 2020-05-21 21:54:41.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 21:59:41.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-21-59-41 patch sysinv 2020-05-21 21:59:41.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:04:41.636 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-04-41 patch sysinv 2020-05-21 22:04:41.636 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:09:41.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-09-41 patch sysinv 2020-05-21 22:09:41.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:14:41.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-14-41 patch sysinv 2020-05-21 22:14:41.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:19:41.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-19-41 patch sysinv 2020-05-21 22:19:41.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:24:41.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-24-41 patch sysinv 2020-05-21 22:24:41.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:29:41.847 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-29-41 patch sysinv 2020-05-21 22:29:41.848 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:34:41.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-34-41 patch sysinv 2020-05-21 22:34:41.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:39:41.919 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-39-41 patch sysinv 2020-05-21 22:39:41.919 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:44:41.835 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-44-41 patch sysinv 2020-05-21 22:44:41.835 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:49:41.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-49-41 patch sysinv 2020-05-21 22:49:41.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:54:41.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-54-41 patch sysinv 2020-05-21 22:54:41.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 22:59:42.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-22-59-41 patch sysinv 2020-05-21 22:59:42.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:04:42.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-04-41 patch sysinv 2020-05-21 23:04:42.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:09:42.182 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-09-42 patch sysinv 2020-05-21 23:09:42.182 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:14:42.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-14-41 patch sysinv 2020-05-21 23:14:42.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:19:42.067 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-19-42 patch sysinv 2020-05-21 23:19:42.068 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:24:42.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-24-42 patch sysinv 2020-05-21 23:24:42.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:29:42.173 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-29-42 patch sysinv 2020-05-21 23:29:42.174 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:34:42.276 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-34-42 patch sysinv 2020-05-21 23:34:42.277 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:39:42.225 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-39-42 patch sysinv 2020-05-21 23:39:42.225 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:44:42.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-44-42 patch sysinv 2020-05-21 23:44:42.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:49:42.271 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-49-42 patch sysinv 2020-05-21 23:49:42.272 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:54:42.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-54-42 patch sysinv 2020-05-21 23:54:42.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-21 23:59:42.349 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-21-23-59-42 patch sysinv 2020-05-21 23:59:42.349 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:04:42.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-04-42 patch sysinv 2020-05-22 00:04:42.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:09:42.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-09-42 patch sysinv 2020-05-22 00:09:42.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:14:42.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-14-42 patch sysinv 2020-05-22 00:14:42.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:19:42.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-19-42 patch sysinv 2020-05-22 00:19:42.464 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:24:42.615 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-24-42 patch sysinv 2020-05-22 00:24:42.616 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:29:42.512 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-29-42 patch sysinv 2020-05-22 00:29:42.513 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:34:42.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-34-42 patch sysinv 2020-05-22 00:34:42.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:39:42.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-39-42 patch sysinv 2020-05-22 00:39:42.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:44:43.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-44-43 patch sysinv 2020-05-22 00:44:43.046 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:49:42.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-49-42 patch sysinv 2020-05-22 00:49:42.589 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:54:42.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-54-42 patch sysinv 2020-05-22 00:54:42.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 00:59:42.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-00-59-42 patch sysinv 2020-05-22 00:59:42.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:04:42.721 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-04-42 patch sysinv 2020-05-22 01:04:42.721 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:09:42.796 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-09-42 patch sysinv 2020-05-22 01:09:42.796 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:14:42.824 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-14-42 patch sysinv 2020-05-22 01:14:42.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:19:42.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-19-42 patch sysinv 2020-05-22 01:19:42.893 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:24:42.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-24-42 patch sysinv 2020-05-22 01:24:42.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:29:42.902 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-29-42 patch sysinv 2020-05-22 01:29:42.903 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:34:42.927 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-34-42 patch sysinv 2020-05-22 01:34:42.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:39:43.042 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-39-43 patch sysinv 2020-05-22 01:39:43.042 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:44:42.938 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-44-42 patch sysinv 2020-05-22 01:44:42.938 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:49:43.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-49-43 patch sysinv 2020-05-22 01:49:43.106 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:54:42.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-54-42 patch sysinv 2020-05-22 01:54:42.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 01:59:43.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-01-59-42 patch sysinv 2020-05-22 01:59:43.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:04:43.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-04-43 patch sysinv 2020-05-22 02:04:43.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:09:43.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-09-43 patch sysinv 2020-05-22 02:09:43.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:14:43.206 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-14-43 patch sysinv 2020-05-22 02:14:43.207 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:19:43.164 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-19-43 patch sysinv 2020-05-22 02:19:43.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:24:43.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-24-43 patch sysinv 2020-05-22 02:24:43.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:29:43.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-29-43 patch sysinv 2020-05-22 02:29:43.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:34:43.392 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-34-43 patch sysinv 2020-05-22 02:34:43.403 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:39:43.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-39-43 patch sysinv 2020-05-22 02:39:43.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:44:43.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-44-43 patch sysinv 2020-05-22 02:44:43.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:49:43.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-49-43 patch sysinv 2020-05-22 02:49:43.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:54:43.507 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-54-43 patch sysinv 2020-05-22 02:54:43.507 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 02:59:43.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-02-59-43 patch sysinv 2020-05-22 02:59:43.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:04:43.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-04-43 patch sysinv 2020-05-22 03:04:43.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:09:43.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-09-43 patch sysinv 2020-05-22 03:09:43.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:14:43.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-14-43 patch sysinv 2020-05-22 03:14:43.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:19:43.542 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-19-43 patch sysinv 2020-05-22 03:19:43.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:24:43.572 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-24-43 patch sysinv 2020-05-22 03:24:43.572 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:29:43.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-29-43 patch sysinv 2020-05-22 03:29:43.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:34:43.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-34-43 patch sysinv 2020-05-22 03:34:43.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:39:43.723 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-39-43 patch sysinv 2020-05-22 03:39:43.724 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:44:43.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-44-43 patch sysinv 2020-05-22 03:44:43.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:49:43.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-49-43 patch sysinv 2020-05-22 03:49:43.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:54:43.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-54-43 patch sysinv 2020-05-22 03:54:43.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 03:59:43.830 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-03-59-43 patch sysinv 2020-05-22 03:59:43.831 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:04:43.874 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-04-43 patch sysinv 2020-05-22 04:04:43.875 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:09:43.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-09-43 patch sysinv 2020-05-22 04:09:43.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:14:43.931 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-14-43 patch sysinv 2020-05-22 04:14:43.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:19:43.904 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-19-43 patch sysinv 2020-05-22 04:19:43.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:24:43.890 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-24-43 patch sysinv 2020-05-22 04:24:43.891 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:29:43.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-29-43 patch sysinv 2020-05-22 04:29:43.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:34:44.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-34-44 patch sysinv 2020-05-22 04:34:44.093 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:39:43.988 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-39-43 patch sysinv 2020-05-22 04:39:43.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:44:44.117 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-44-44 patch sysinv 2020-05-22 04:44:44.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:49:44.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-49-44 patch sysinv 2020-05-22 04:49:44.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:54:44.216 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-54-44 patch sysinv 2020-05-22 04:54:44.216 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 04:59:44.162 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-04-59-44 patch sysinv 2020-05-22 04:59:44.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:04:44.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-04-44 patch sysinv 2020-05-22 05:04:44.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:09:44.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-09-44 patch sysinv 2020-05-22 05:09:44.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:14:44.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-14-44 patch sysinv 2020-05-22 05:14:44.382 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:19:44.274 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-19-44 patch sysinv 2020-05-22 05:19:44.274 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:24:44.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-24-44 patch sysinv 2020-05-22 05:24:44.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:29:44.394 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-29-44 patch sysinv 2020-05-22 05:29:44.395 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:34:44.406 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-34-44 patch sysinv 2020-05-22 05:34:44.407 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:39:44.421 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-39-44 patch sysinv 2020-05-22 05:39:44.421 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:44:44.452 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-44-44 patch sysinv 2020-05-22 05:44:44.453 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:49:44.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-49-44 patch sysinv 2020-05-22 05:49:44.547 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:54:44.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-54-44 patch sysinv 2020-05-22 05:54:44.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 05:59:44.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-05-59-44 patch sysinv 2020-05-22 05:59:44.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:04:44.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-04-44 patch sysinv 2020-05-22 06:04:44.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:09:44.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-09-44 patch sysinv 2020-05-22 06:09:44.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:14:44.652 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-14-44 patch sysinv 2020-05-22 06:14:44.652 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:19:44.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-19-44 patch sysinv 2020-05-22 06:19:44.729 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:24:44.696 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-24-44 patch sysinv 2020-05-22 06:24:44.696 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:29:44.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-29-44 patch sysinv 2020-05-22 06:29:44.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:34:44.757 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-34-44 patch sysinv 2020-05-22 06:34:44.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:39:44.848 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-39-44 patch sysinv 2020-05-22 06:39:44.849 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:44:45.192 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-44-45 patch sysinv 2020-05-22 06:44:45.193 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:49:44.773 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-49-44 patch sysinv 2020-05-22 06:49:44.774 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:54:44.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-54-44 patch sysinv 2020-05-22 06:54:44.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 06:59:44.832 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-06-59-44 patch sysinv 2020-05-22 06:59:44.832 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:04:44.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-04-44 patch sysinv 2020-05-22 07:04:44.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:09:44.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-09-44 patch sysinv 2020-05-22 07:09:44.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:14:45.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-14-45 patch sysinv 2020-05-22 07:14:45.076 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:19:44.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-19-44 patch sysinv 2020-05-22 07:19:44.956 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:24:44.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-24-44 patch sysinv 2020-05-22 07:24:44.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:29:45.164 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-29-45 patch sysinv 2020-05-22 07:29:45.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:34:45.169 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-34-45 patch sysinv 2020-05-22 07:34:45.169 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:39:45.244 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-39-45 patch sysinv 2020-05-22 07:39:45.244 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:44:45.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-44-45 patch sysinv 2020-05-22 07:44:45.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:49:45.216 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-49-45 patch sysinv 2020-05-22 07:49:45.217 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:54:45.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-54-45 patch sysinv 2020-05-22 07:54:45.310 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 07:59:45.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-07-59-45 patch sysinv 2020-05-22 07:59:45.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:04:45.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-04-45 patch sysinv 2020-05-22 08:04:45.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:09:45.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-09-45 patch sysinv 2020-05-22 08:09:45.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:14:45.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-14-45 patch sysinv 2020-05-22 08:14:45.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:19:45.432 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-19-45 patch sysinv 2020-05-22 08:19:45.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:24:45.387 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-24-45 patch sysinv 2020-05-22 08:24:45.388 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:29:45.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-29-45 patch sysinv 2020-05-22 08:29:45.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:34:45.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-34-45 patch sysinv 2020-05-22 08:34:45.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:39:45.585 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-39-45 patch sysinv 2020-05-22 08:39:45.586 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:44:45.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-44-45 patch sysinv 2020-05-22 08:44:45.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:49:45.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-49-45 patch sysinv 2020-05-22 08:49:45.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:54:45.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-54-45 patch sysinv 2020-05-22 08:54:45.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 08:59:45.686 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-08-59-45 patch sysinv 2020-05-22 08:59:45.686 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:04:45.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-04-45 patch sysinv 2020-05-22 09:04:45.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:09:45.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-09-45 patch sysinv 2020-05-22 09:09:45.770 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:14:45.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-14-45 patch sysinv 2020-05-22 09:14:45.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:19:45.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-19-45 patch sysinv 2020-05-22 09:19:45.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:24:45.773 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-24-45 patch sysinv 2020-05-22 09:24:45.774 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:29:45.888 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-29-45 patch sysinv 2020-05-22 09:29:45.888 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:34:45.875 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-34-45 patch sysinv 2020-05-22 09:34:45.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:39:45.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-39-45 patch sysinv 2020-05-22 09:39:45.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:44:45.879 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-44-45 patch sysinv 2020-05-22 09:44:45.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:49:46.021 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-49-45 patch sysinv 2020-05-22 09:49:46.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:54:45.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-54-45 patch sysinv 2020-05-22 09:54:45.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 09:59:46.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-09-59-46 patch sysinv 2020-05-22 09:59:46.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:04:45.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-04-45 patch sysinv 2020-05-22 10:04:45.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:09:46.142 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-09-46 patch sysinv 2020-05-22 10:09:46.142 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:14:46.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-14-45 patch sysinv 2020-05-22 10:14:46.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:19:46.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-19-46 patch sysinv 2020-05-22 10:19:46.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:24:46.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-24-46 patch sysinv 2020-05-22 10:24:46.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:29:46.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-29-46 patch sysinv 2020-05-22 10:29:46.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:34:46.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-34-46 patch sysinv 2020-05-22 10:34:46.356 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:39:46.281 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-39-46 patch sysinv 2020-05-22 10:39:46.282 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:44:46.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-44-46 patch sysinv 2020-05-22 10:44:46.756 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:49:46.304 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-49-46 patch sysinv 2020-05-22 10:49:46.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:54:46.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-54-46 patch sysinv 2020-05-22 10:54:46.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 10:59:46.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-10-59-46 patch sysinv 2020-05-22 10:59:46.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:04:46.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-04-46 patch sysinv 2020-05-22 11:04:46.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:09:46.444 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-09-46 patch sysinv 2020-05-22 11:09:46.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:14:46.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-14-46 patch sysinv 2020-05-22 11:14:46.458 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:19:46.565 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-19-46 patch sysinv 2020-05-22 11:19:46.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:24:46.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-24-46 patch sysinv 2020-05-22 11:24:46.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:29:46.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-29-46 patch sysinv 2020-05-22 11:29:46.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:34:46.570 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-34-46 patch sysinv 2020-05-22 11:34:46.570 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:39:46.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-39-46 patch sysinv 2020-05-22 11:39:46.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:44:46.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-44-46 patch sysinv 2020-05-22 11:44:46.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:49:46.660 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-49-46 patch sysinv 2020-05-22 11:49:46.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:54:46.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-54-46 patch sysinv 2020-05-22 11:54:46.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 11:59:46.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-11-59-46 patch sysinv 2020-05-22 11:59:46.708 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:04:46.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-04-46 patch sysinv 2020-05-22 12:04:46.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:09:46.768 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-09-46 patch sysinv 2020-05-22 12:09:46.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:14:46.855 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-14-46 patch sysinv 2020-05-22 12:14:46.855 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:19:46.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-19-46 patch sysinv 2020-05-22 12:19:46.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:24:47.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-24-47 patch sysinv 2020-05-22 12:24:47.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:29:46.848 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-29-46 patch sysinv 2020-05-22 12:29:46.848 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:34:46.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-34-46 patch sysinv 2020-05-22 12:34:46.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:39:46.903 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-39-46 patch sysinv 2020-05-22 12:39:46.903 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:44:46.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-44-46 patch sysinv 2020-05-22 12:44:46.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:49:47.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-49-47 patch sysinv 2020-05-22 12:49:47.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:54:46.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-54-46 patch sysinv 2020-05-22 12:54:46.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 12:59:47.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-12-59-46 patch sysinv 2020-05-22 12:59:47.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:04:47.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-04-47 patch sysinv 2020-05-22 13:04:47.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:09:47.189 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-09-47 patch sysinv 2020-05-22 13:09:47.190 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:14:47.225 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-14-47 patch sysinv 2020-05-22 13:14:47.225 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:19:47.252 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-19-47 patch sysinv 2020-05-22 13:19:47.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:24:47.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-24-47 patch sysinv 2020-05-22 13:24:47.342 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:29:47.310 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-29-47 patch sysinv 2020-05-22 13:29:47.311 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:34:47.342 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-34-47 patch sysinv 2020-05-22 13:34:47.343 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:39:47.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-39-47 patch sysinv 2020-05-22 13:39:47.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:44:47.401 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-44-47 patch sysinv 2020-05-22 13:44:47.402 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:49:47.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-49-47 patch sysinv 2020-05-22 13:49:47.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:54:47.483 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-54-47 patch sysinv 2020-05-22 13:54:47.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 13:59:47.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-13-59-47 patch sysinv 2020-05-22 13:59:47.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:04:47.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-04-47 patch sysinv 2020-05-22 14:04:47.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:09:47.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-09-47 patch sysinv 2020-05-22 14:09:47.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:14:47.617 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-14-47 patch sysinv 2020-05-22 14:14:47.617 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:19:47.628 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-19-47 patch sysinv 2020-05-22 14:19:47.629 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:24:47.704 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-24-47 patch sysinv 2020-05-22 14:24:47.704 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:29:47.615 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-29-47 patch sysinv 2020-05-22 14:29:47.615 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:34:47.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-34-47 patch sysinv 2020-05-22 14:34:47.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:39:47.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-39-47 patch sysinv 2020-05-22 14:39:47.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:44:48.224 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-44-48 patch sysinv 2020-05-22 14:44:48.224 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:49:47.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-49-47 patch sysinv 2020-05-22 14:49:47.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:54:47.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-54-47 patch sysinv 2020-05-22 14:54:47.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 14:59:47.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-14-59-47 patch sysinv 2020-05-22 14:59:47.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:04:47.822 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-04-47 patch sysinv 2020-05-22 15:04:47.823 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:09:47.944 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-09-47 patch sysinv 2020-05-22 15:09:47.945 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:14:47.898 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-14-47 patch sysinv 2020-05-22 15:14:47.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:19:47.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-19-47 patch sysinv 2020-05-22 15:19:47.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:24:47.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-24-47 patch sysinv 2020-05-22 15:24:47.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:29:48.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-29-47 patch sysinv 2020-05-22 15:29:48.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:34:48.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-34-48 patch sysinv 2020-05-22 15:34:48.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:39:48.175 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-39-48 patch sysinv 2020-05-22 15:39:48.175 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:44:48.432 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-44-48 patch sysinv 2020-05-22 15:44:48.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:49:48.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-49-48 patch sysinv 2020-05-22 15:49:48.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:54:48.211 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-54-48 patch sysinv 2020-05-22 15:54:48.211 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 15:59:48.186 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-15-59-48 patch sysinv 2020-05-22 15:59:48.187 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:04:48.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-04-48 patch sysinv 2020-05-22 16:04:48.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:09:48.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-09-48 patch sysinv 2020-05-22 16:09:48.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:14:48.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-14-48 patch sysinv 2020-05-22 16:14:48.271 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:19:48.358 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-19-48 patch sysinv 2020-05-22 16:19:48.359 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:24:48.399 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-24-48 patch sysinv 2020-05-22 16:24:48.399 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:29:48.406 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-29-48 patch sysinv 2020-05-22 16:29:48.407 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:34:48.428 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-34-48 patch sysinv 2020-05-22 16:34:48.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:39:48.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-39-48 patch sysinv 2020-05-22 16:39:48.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:44:48.856 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-44-48 patch sysinv 2020-05-22 16:44:48.856 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:49:48.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-49-48 patch sysinv 2020-05-22 16:49:48.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:54:48.607 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-54-48 patch sysinv 2020-05-22 16:54:48.608 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 16:59:48.670 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-16-59-48 patch sysinv 2020-05-22 16:59:48.671 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:04:48.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-04-48 patch sysinv 2020-05-22 17:04:48.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:09:48.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-09-48 patch sysinv 2020-05-22 17:09:48.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:14:48.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-14-48 patch sysinv 2020-05-22 17:14:48.670 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:19:48.820 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-19-48 patch sysinv 2020-05-22 17:19:48.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:24:48.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-24-48 patch sysinv 2020-05-22 17:24:48.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:29:48.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-29-48 patch sysinv 2020-05-22 17:29:48.744 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:34:48.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-34-48 patch sysinv 2020-05-22 17:34:48.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:39:48.782 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-39-48 patch sysinv 2020-05-22 17:39:48.782 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:44:49.158 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-44-49 patch sysinv 2020-05-22 17:44:49.158 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:49:48.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-49-48 patch sysinv 2020-05-22 17:49:48.794 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:54:48.826 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-54-48 patch sysinv 2020-05-22 17:54:48.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 17:59:48.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-17-59-48 patch sysinv 2020-05-22 17:59:48.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:04:48.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-04-48 patch sysinv 2020-05-22 18:04:48.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:09:49.025 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-09-49 patch sysinv 2020-05-22 18:09:49.025 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:14:48.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-14-48 patch sysinv 2020-05-22 18:14:48.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:19:48.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-19-48 patch sysinv 2020-05-22 18:19:48.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:24:49.061 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-24-49 patch sysinv 2020-05-22 18:24:49.061 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:29:49.078 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-29-49 patch sysinv 2020-05-22 18:29:49.078 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:34:49.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-34-49 patch sysinv 2020-05-22 18:34:49.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:39:49.137 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-39-49 patch sysinv 2020-05-22 18:39:49.138 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:44:49.128 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-44-49 patch sysinv 2020-05-22 18:44:49.129 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:49:49.226 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-49-49 patch sysinv 2020-05-22 18:49:49.227 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:54:49.263 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-54-49 patch sysinv 2020-05-22 18:54:49.263 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 18:59:49.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-18-59-49 patch sysinv 2020-05-22 18:59:49.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:04:49.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-04-49 patch sysinv 2020-05-22 19:04:49.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:09:49.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-09-49 patch sysinv 2020-05-22 19:09:49.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:14:49.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-14-49 patch sysinv 2020-05-22 19:14:49.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:19:49.432 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-19-49 patch sysinv 2020-05-22 19:19:49.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:24:49.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-24-49 patch sysinv 2020-05-22 19:24:49.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:29:49.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-29-49 patch sysinv 2020-05-22 19:29:49.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:34:49.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-34-49 patch sysinv 2020-05-22 19:34:49.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:39:49.630 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-39-49 patch sysinv 2020-05-22 19:39:49.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:44:49.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-44-49 patch sysinv 2020-05-22 19:44:49.969 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:49:49.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-49-49 patch sysinv 2020-05-22 19:49:49.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:54:49.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-54-49 patch sysinv 2020-05-22 19:54:49.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 19:59:49.780 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-19-59-49 patch sysinv 2020-05-22 19:59:49.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:04:49.677 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-04-49 patch sysinv 2020-05-22 20:04:49.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:09:49.845 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-09-49 patch sysinv 2020-05-22 20:09:49.845 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:14:49.773 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-14-49 patch sysinv 2020-05-22 20:14:49.774 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:19:49.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-19-49 patch sysinv 2020-05-22 20:19:49.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:24:49.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-24-49 patch sysinv 2020-05-22 20:24:49.786 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:29:49.826 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-29-49 patch sysinv 2020-05-22 20:29:49.826 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:34:49.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-34-49 patch sysinv 2020-05-22 20:34:49.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:39:49.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-39-49 patch sysinv 2020-05-22 20:39:49.912 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:44:50.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-44-50 patch sysinv 2020-05-22 20:44:50.390 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:49:49.887 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-49-49 patch sysinv 2020-05-22 20:49:49.887 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:54:49.929 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-54-49 patch sysinv 2020-05-22 20:54:49.930 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 20:59:50.093 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-20-59-50 patch sysinv 2020-05-22 20:59:50.093 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:04:50.041 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-04-50 patch sysinv 2020-05-22 21:04:50.042 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:09:50.026 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-09-49 patch sysinv 2020-05-22 21:09:50.026 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:14:50.055 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-14-50 patch sysinv 2020-05-22 21:14:50.056 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:19:50.082 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-19-50 patch sysinv 2020-05-22 21:19:50.082 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:24:50.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-24-50 patch sysinv 2020-05-22 21:24:50.164 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:29:50.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-29-50 patch sysinv 2020-05-22 21:29:50.298 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:34:50.346 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-34-50 patch sysinv 2020-05-22 21:34:50.346 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:39:50.263 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-39-50 patch sysinv 2020-05-22 21:39:50.264 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:44:50.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-44-50 patch sysinv 2020-05-22 21:44:50.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:49:50.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-49-50 patch sysinv 2020-05-22 21:49:50.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:54:50.462 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-54-50 patch sysinv 2020-05-22 21:54:50.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 21:59:50.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-21-59-50 patch sysinv 2020-05-22 21:59:50.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:04:50.354 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-04-50 patch sysinv 2020-05-22 22:04:50.354 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:09:50.524 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-09-50 patch sysinv 2020-05-22 22:09:50.525 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:14:50.519 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-14-50 patch sysinv 2020-05-22 22:14:50.519 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:19:50.576 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-19-50 patch sysinv 2020-05-22 22:19:50.577 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:24:50.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-24-50 patch sysinv 2020-05-22 22:24:50.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:29:50.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-29-50 patch sysinv 2020-05-22 22:29:50.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:34:50.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-34-50 patch sysinv 2020-05-22 22:34:50.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:39:50.701 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-39-50 patch sysinv 2020-05-22 22:39:50.702 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:44:50.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-44-50 patch sysinv 2020-05-22 22:44:50.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:49:50.737 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-49-50 patch sysinv 2020-05-22 22:49:50.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:54:50.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-54-50 patch sysinv 2020-05-22 22:54:50.719 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 22:59:50.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-22-59-50 patch sysinv 2020-05-22 22:59:50.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:04:50.787 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-04-50 patch sysinv 2020-05-22 23:04:50.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:09:50.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-09-50 patch sysinv 2020-05-22 23:09:50.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:14:50.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-14-50 patch sysinv 2020-05-22 23:14:50.890 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:19:51.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-19-50 patch sysinv 2020-05-22 23:19:51.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:24:50.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-24-50 patch sysinv 2020-05-22 23:24:50.900 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:29:51.090 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-29-51 patch sysinv 2020-05-22 23:29:51.091 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:34:50.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-34-50 patch sysinv 2020-05-22 23:34:50.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:39:50.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-39-50 patch sysinv 2020-05-22 23:39:50.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:44:51.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-44-51 patch sysinv 2020-05-22 23:44:51.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:49:50.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-49-50 patch sysinv 2020-05-22 23:49:50.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:54:51.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-54-50 patch sysinv 2020-05-22 23:54:51.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-22 23:59:51.116 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-22-23-59-51 patch sysinv 2020-05-22 23:59:51.116 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:04:51.231 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-04-51 patch sysinv 2020-05-23 00:04:51.231 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:09:51.141 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-09-51 patch sysinv 2020-05-23 00:09:51.142 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:14:51.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-14-51 patch sysinv 2020-05-23 00:14:51.294 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:19:51.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-19-51 patch sysinv 2020-05-23 00:19:51.203 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:24:51.330 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-24-51 patch sysinv 2020-05-23 00:24:51.331 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:29:51.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-29-51 patch sysinv 2020-05-23 00:29:51.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:34:51.408 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-34-51 patch sysinv 2020-05-23 00:34:51.409 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:39:51.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-39-51 patch sysinv 2020-05-23 00:39:51.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:44:51.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-44-51 patch sysinv 2020-05-23 00:44:51.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:49:51.410 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-49-51 patch sysinv 2020-05-23 00:49:51.410 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:54:51.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-54-51 patch sysinv 2020-05-23 00:54:51.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 00:59:51.582 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-00-59-51 patch sysinv 2020-05-23 00:59:51.582 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:04:51.452 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-04-51 patch sysinv 2020-05-23 01:04:51.453 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:09:51.642 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-09-51 patch sysinv 2020-05-23 01:09:51.642 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:14:51.593 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-14-51 patch sysinv 2020-05-23 01:14:51.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:19:51.573 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-19-51 patch sysinv 2020-05-23 01:19:51.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:24:51.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-24-51 patch sysinv 2020-05-23 01:24:51.567 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:29:51.757 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-29-51 patch sysinv 2020-05-23 01:29:51.757 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:34:51.756 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-34-51 patch sysinv 2020-05-23 01:34:51.756 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:39:52.117 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-39-52 patch sysinv 2020-05-23 01:39:52.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:44:51.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-44-51 patch sysinv 2020-05-23 01:44:51.693 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:49:51.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-49-51 patch sysinv 2020-05-23 01:49:51.874 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:54:51.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-54-51 patch sysinv 2020-05-23 01:54:51.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 01:59:51.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-01-59-51 patch sysinv 2020-05-23 01:59:51.837 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:04:51.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-04-51 patch sysinv 2020-05-23 02:04:51.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:09:51.897 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-09-51 patch sysinv 2020-05-23 02:09:51.898 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:14:51.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-14-51 patch sysinv 2020-05-23 02:14:51.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:19:51.952 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-19-51 patch sysinv 2020-05-23 02:19:51.952 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:24:52.223 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-24-52 patch sysinv 2020-05-23 02:24:52.224 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:29:51.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-29-51 patch sysinv 2020-05-23 02:29:51.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:34:51.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-34-51 patch sysinv 2020-05-23 02:34:51.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:39:52.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-39-52 patch sysinv 2020-05-23 02:39:52.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:44:52.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-44-52 patch sysinv 2020-05-23 02:44:52.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:49:52.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-49-52 patch sysinv 2020-05-23 02:49:52.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:54:52.251 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-54-52 patch sysinv 2020-05-23 02:54:52.252 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 02:59:52.212 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-02-59-52 patch sysinv 2020-05-23 02:59:52.213 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:04:52.282 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-04-52 patch sysinv 2020-05-23 03:04:52.283 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:09:52.249 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-09-52 patch sysinv 2020-05-23 03:09:52.250 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:14:52.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-14-52 patch sysinv 2020-05-23 03:14:52.367 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:19:52.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-19-52 patch sysinv 2020-05-23 03:19:52.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:24:52.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-24-52 patch sysinv 2020-05-23 03:24:52.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:29:52.427 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-29-52 patch sysinv 2020-05-23 03:29:52.427 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:34:52.496 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-34-52 patch sysinv 2020-05-23 03:34:52.497 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:39:52.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-39-52 patch sysinv 2020-05-23 03:39:52.877 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:44:52.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-44-52 patch sysinv 2020-05-23 03:44:52.518 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:49:52.511 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-49-52 patch sysinv 2020-05-23 03:49:52.512 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:54:52.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-54-52 patch sysinv 2020-05-23 03:54:52.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 03:59:52.548 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-03-59-52 patch sysinv 2020-05-23 03:59:52.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:04:52.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-04-52 patch sysinv 2020-05-23 04:04:52.627 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:09:52.638 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-09-52 patch sysinv 2020-05-23 04:09:52.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:14:52.831 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-14-52 patch sysinv 2020-05-23 04:14:52.831 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:19:52.698 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-19-52 patch sysinv 2020-05-23 04:19:52.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:24:52.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-24-52 patch sysinv 2020-05-23 04:24:52.748 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:29:52.835 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-29-52 patch sysinv 2020-05-23 04:29:52.835 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:34:52.824 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-34-52 patch sysinv 2020-05-23 04:34:52.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:39:52.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-39-52 patch sysinv 2020-05-23 04:39:52.874 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:44:52.786 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-44-52 patch sysinv 2020-05-23 04:44:52.787 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:49:52.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-49-52 patch sysinv 2020-05-23 04:49:52.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:54:52.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-54-52 patch sysinv 2020-05-23 04:54:52.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 04:59:53.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-04-59-53 patch sysinv 2020-05-23 04:59:53.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:04:52.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-04-52 patch sysinv 2020-05-23 05:04:52.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:09:52.998 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-09-52 patch sysinv 2020-05-23 05:09:52.999 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:14:53.112 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-14-53 patch sysinv 2020-05-23 05:14:53.112 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:19:53.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-19-52 patch sysinv 2020-05-23 05:19:53.011 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:24:53.178 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-24-53 patch sysinv 2020-05-23 05:24:53.179 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:29:53.120 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-29-53 patch sysinv 2020-05-23 05:29:53.121 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:34:53.191 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-34-53 patch sysinv 2020-05-23 05:34:53.192 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:39:53.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-39-53 patch sysinv 2020-05-23 05:39:53.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:44:53.383 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-44-53 patch sysinv 2020-05-23 05:44:53.384 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:49:53.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-49-53 patch sysinv 2020-05-23 05:49:53.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:54:53.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-54-53 patch sysinv 2020-05-23 05:54:53.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 05:59:53.360 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-05-59-53 patch sysinv 2020-05-23 05:59:53.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:04:53.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-04-53 patch sysinv 2020-05-23 06:04:53.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:09:53.432 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-09-53 patch sysinv 2020-05-23 06:09:53.432 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:14:53.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-14-53 patch sysinv 2020-05-23 06:14:53.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:19:53.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-19-53 patch sysinv 2020-05-23 06:19:53.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:24:53.465 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-24-53 patch sysinv 2020-05-23 06:24:53.465 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:29:53.568 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-29-53 patch sysinv 2020-05-23 06:29:53.568 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:34:53.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-34-53 patch sysinv 2020-05-23 06:34:53.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:39:54.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-39-54 patch sysinv 2020-05-23 06:39:54.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:44:53.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-44-53 patch sysinv 2020-05-23 06:44:53.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:49:53.616 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-49-53 patch sysinv 2020-05-23 06:49:53.617 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:54:53.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-54-53 patch sysinv 2020-05-23 06:54:53.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 06:59:53.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-06-59-53 patch sysinv 2020-05-23 06:59:53.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:04:53.746 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-04-53 patch sysinv 2020-05-23 07:04:53.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:09:53.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-09-53 patch sysinv 2020-05-23 07:09:53.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:14:53.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-14-53 patch sysinv 2020-05-23 07:14:53.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:19:53.802 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-19-53 patch sysinv 2020-05-23 07:19:53.803 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:24:53.977 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-24-53 patch sysinv 2020-05-23 07:24:53.978 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:29:53.845 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-29-53 patch sysinv 2020-05-23 07:29:53.846 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:34:54.089 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-34-54 patch sysinv 2020-05-23 07:34:54.090 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:39:53.860 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-39-53 patch sysinv 2020-05-23 07:39:53.861 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:44:53.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-44-53 patch sysinv 2020-05-23 07:44:53.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:49:53.999 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-49-53 patch sysinv 2020-05-23 07:49:53.999 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:54:54.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-54-53 patch sysinv 2020-05-23 07:54:54.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 07:59:54.063 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-07-59-54 patch sysinv 2020-05-23 07:59:54.063 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:04:54.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-04-53 patch sysinv 2020-05-23 08:04:54.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:09:54.306 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-09-54 patch sysinv 2020-05-23 08:09:54.306 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:14:54.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-14-54 patch sysinv 2020-05-23 08:14:54.157 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:19:54.384 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-19-54 patch sysinv 2020-05-23 08:19:54.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:24:54.210 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-24-54 patch sysinv 2020-05-23 08:24:54.211 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:29:54.204 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-29-54 patch sysinv 2020-05-23 08:29:54.205 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:34:54.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-34-54 patch sysinv 2020-05-23 08:34:54.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:39:54.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-39-54 patch sysinv 2020-05-23 08:39:54.644 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:44:54.364 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-44-54 patch sysinv 2020-05-23 08:44:54.364 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:49:54.334 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-49-54 patch sysinv 2020-05-23 08:49:54.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:54:54.398 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-54-54 patch sysinv 2020-05-23 08:54:54.398 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 08:59:54.379 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-08-59-54 patch sysinv 2020-05-23 08:59:54.379 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:04:54.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-04-54 patch sysinv 2020-05-23 09:04:54.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:09:54.434 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-09-54 patch sysinv 2020-05-23 09:09:54.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:14:54.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-14-54 patch sysinv 2020-05-23 09:14:54.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:19:54.501 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-19-54 patch sysinv 2020-05-23 09:19:54.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:24:54.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-24-54 patch sysinv 2020-05-23 09:24:54.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:29:54.612 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-29-54 patch sysinv 2020-05-23 09:29:54.613 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:34:54.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-34-54 patch sysinv 2020-05-23 09:34:54.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:39:54.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-39-54 patch sysinv 2020-05-23 09:39:54.620 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:44:54.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-44-54 patch sysinv 2020-05-23 09:44:54.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:49:54.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-49-54 patch sysinv 2020-05-23 09:49:54.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:54:54.804 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-54-54 patch sysinv 2020-05-23 09:54:54.804 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 09:59:54.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-09-59-54 patch sysinv 2020-05-23 09:59:54.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:04:54.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-04-54 patch sysinv 2020-05-23 10:04:54.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:09:54.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-09-54 patch sysinv 2020-05-23 10:09:54.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:14:54.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-14-54 patch sysinv 2020-05-23 10:14:54.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:19:54.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-19-54 patch sysinv 2020-05-23 10:19:54.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:24:54.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-24-54 patch sysinv 2020-05-23 10:24:54.958 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:29:55.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-29-55 patch sysinv 2020-05-23 10:29:55.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:34:54.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-34-54 patch sysinv 2020-05-23 10:34:54.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:39:54.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-39-54 patch sysinv 2020-05-23 10:39:54.958 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:44:54.976 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-44-54 patch sysinv 2020-05-23 10:44:54.977 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:49:55.100 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-49-55 patch sysinv 2020-05-23 10:49:55.100 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:54:55.175 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-54-55 patch sysinv 2020-05-23 10:54:55.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 10:59:55.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-10-59-55 patch sysinv 2020-05-23 10:59:55.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:04:55.194 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-04-55 patch sysinv 2020-05-23 11:04:55.195 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:09:55.255 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-09-55 patch sysinv 2020-05-23 11:09:55.255 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:14:55.268 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-14-55 patch sysinv 2020-05-23 11:14:55.269 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:19:55.249 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-19-55 patch sysinv 2020-05-23 11:19:55.249 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:24:55.307 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-24-55 patch sysinv 2020-05-23 11:24:55.307 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:29:55.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-29-55 patch sysinv 2020-05-23 11:29:55.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:34:55.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-34-55 patch sysinv 2020-05-23 11:34:55.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:39:55.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-39-55 patch sysinv 2020-05-23 11:39:55.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:44:55.428 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-44-55 patch sysinv 2020-05-23 11:44:55.428 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:49:55.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-49-55 patch sysinv 2020-05-23 11:49:55.470 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:54:55.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-54-55 patch sysinv 2020-05-23 11:54:55.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 11:59:55.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-11-59-55 patch sysinv 2020-05-23 11:59:55.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:04:55.550 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-04-55 patch sysinv 2020-05-23 12:04:55.550 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:09:55.579 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-09-55 patch sysinv 2020-05-23 12:09:55.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:14:55.628 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-14-55 patch sysinv 2020-05-23 12:14:55.628 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:19:55.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-19-55 patch sysinv 2020-05-23 12:19:55.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:24:55.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-24-55 patch sysinv 2020-05-23 12:24:55.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:29:55.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-29-55 patch sysinv 2020-05-23 12:29:55.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:34:56.171 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-34-56 patch sysinv 2020-05-23 12:34:56.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:39:55.705 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-39-55 patch sysinv 2020-05-23 12:39:55.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:44:55.779 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-44-55 patch sysinv 2020-05-23 12:44:55.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:49:55.839 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-49-55 patch sysinv 2020-05-23 12:49:55.839 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:54:55.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-54-55 patch sysinv 2020-05-23 12:54:55.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 12:59:55.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-12-59-55 patch sysinv 2020-05-23 12:59:55.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:04:55.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-04-55 patch sysinv 2020-05-23 13:04:55.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:09:55.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-09-55 patch sysinv 2020-05-23 13:09:55.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:14:56.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-14-56 patch sysinv 2020-05-23 13:14:56.076 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:19:55.918 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-19-55 patch sysinv 2020-05-23 13:19:55.919 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:24:56.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-24-56 patch sysinv 2020-05-23 13:24:56.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:29:55.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-29-55 patch sysinv 2020-05-23 13:29:55.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:34:56.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-34-55 patch sysinv 2020-05-23 13:34:56.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:39:56.064 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-39-56 patch sysinv 2020-05-23 13:39:56.065 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:44:56.113 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-44-56 patch sysinv 2020-05-23 13:44:56.113 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:49:56.210 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-49-56 patch sysinv 2020-05-23 13:49:56.212 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:54:56.237 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-54-56 patch sysinv 2020-05-23 13:54:56.238 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 13:59:56.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-13-59-56 patch sysinv 2020-05-23 13:59:56.390 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:04:56.295 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-04-56 patch sysinv 2020-05-23 14:04:56.296 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:09:56.416 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-09-56 patch sysinv 2020-05-23 14:09:56.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:14:56.384 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-14-56 patch sysinv 2020-05-23 14:14:56.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:19:56.344 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-19-56 patch sysinv 2020-05-23 14:19:56.345 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:24:56.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-24-56 patch sysinv 2020-05-23 14:24:56.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:29:56.421 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-29-56 patch sysinv 2020-05-23 14:29:56.422 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:34:56.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-34-56 patch sysinv 2020-05-23 14:34:56.458 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:39:56.459 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-39-56 patch sysinv 2020-05-23 14:39:56.460 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:44:56.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-44-56 patch sysinv 2020-05-23 14:44:56.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:49:56.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-49-56 patch sysinv 2020-05-23 14:49:56.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:54:56.557 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-54-56 patch sysinv 2020-05-23 14:54:56.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 14:59:56.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-14-59-56 patch sysinv 2020-05-23 14:59:56.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:04:56.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-04-56 patch sysinv 2020-05-23 15:04:56.601 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:09:56.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-09-56 patch sysinv 2020-05-23 15:09:56.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:14:56.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-14-56 patch sysinv 2020-05-23 15:14:56.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:19:56.722 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-19-56 patch sysinv 2020-05-23 15:19:56.723 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:24:56.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-24-56 patch sysinv 2020-05-23 15:24:56.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:29:56.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-29-56 patch sysinv 2020-05-23 15:29:56.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:34:57.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-34-57 patch sysinv 2020-05-23 15:34:57.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:39:56.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-39-56 patch sysinv 2020-05-23 15:39:56.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:44:56.851 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-44-56 patch sysinv 2020-05-23 15:44:56.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:49:56.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-49-56 patch sysinv 2020-05-23 15:49:56.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:54:56.963 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-54-56 patch sysinv 2020-05-23 15:54:56.963 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 15:59:56.945 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-15-59-56 patch sysinv 2020-05-23 15:59:56.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:04:57.067 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-04-57 patch sysinv 2020-05-23 16:04:57.067 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:09:56.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-09-56 patch sysinv 2020-05-23 16:09:56.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:14:56.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-14-56 patch sysinv 2020-05-23 16:14:56.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:19:57.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-19-57 patch sysinv 2020-05-23 16:19:57.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:24:57.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-24-57 patch sysinv 2020-05-23 16:24:57.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:29:57.189 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-29-57 patch sysinv 2020-05-23 16:29:57.190 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:34:57.601 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-34-57 patch sysinv 2020-05-23 16:34:57.602 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:39:57.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-39-57 patch sysinv 2020-05-23 16:39:57.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:44:57.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-44-57 patch sysinv 2020-05-23 16:44:57.203 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:49:57.277 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-49-57 patch sysinv 2020-05-23 16:49:57.278 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:54:57.356 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-54-57 patch sysinv 2020-05-23 16:54:57.357 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 16:59:57.349 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-16-59-57 patch sysinv 2020-05-23 16:59:57.349 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:04:57.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-04-57 patch sysinv 2020-05-23 17:04:57.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:09:57.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-09-57 patch sysinv 2020-05-23 17:09:57.504 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:14:57.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-14-57 patch sysinv 2020-05-23 17:14:57.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:19:57.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-19-57 patch sysinv 2020-05-23 17:19:57.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:24:57.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-24-57 patch sysinv 2020-05-23 17:24:57.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:29:57.491 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-29-57 patch sysinv 2020-05-23 17:29:57.492 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:34:57.633 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-34-57 patch sysinv 2020-05-23 17:34:57.634 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:39:57.572 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-39-57 patch sysinv 2020-05-23 17:39:57.573 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:44:57.553 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-44-57 patch sysinv 2020-05-23 17:44:57.553 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:49:57.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-49-57 patch sysinv 2020-05-23 17:49:57.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:54:57.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-54-57 patch sysinv 2020-05-23 17:54:57.644 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 17:59:57.749 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-17-59-57 patch sysinv 2020-05-23 17:59:57.750 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:04:57.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-04-57 patch sysinv 2020-05-23 18:04:57.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:09:57.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-09-57 patch sysinv 2020-05-23 18:09:57.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:14:57.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-14-57 patch sysinv 2020-05-23 18:14:57.807 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:19:57.790 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-19-57 patch sysinv 2020-05-23 18:19:57.791 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:24:57.878 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-24-57 patch sysinv 2020-05-23 18:24:57.879 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:29:57.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-29-57 patch sysinv 2020-05-23 18:29:57.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:34:57.923 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-34-57 patch sysinv 2020-05-23 18:34:57.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:39:57.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-39-57 patch sysinv 2020-05-23 18:39:57.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:44:58.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-44-57 patch sysinv 2020-05-23 18:44:58.020 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:49:58.103 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-49-58 patch sysinv 2020-05-23 18:49:58.104 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:54:57.955 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-54-57 patch sysinv 2020-05-23 18:54:57.956 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 18:59:57.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-18-59-57 patch sysinv 2020-05-23 18:59:57.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:04:58.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-04-58 patch sysinv 2020-05-23 19:04:58.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:09:58.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-09-58 patch sysinv 2020-05-23 19:09:58.186 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:14:58.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-14-58 patch sysinv 2020-05-23 19:14:58.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:19:58.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-19-58 patch sysinv 2020-05-23 19:19:58.171 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:24:58.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-24-58 patch sysinv 2020-05-23 19:24:58.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:29:58.203 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-29-58 patch sysinv 2020-05-23 19:29:58.204 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:34:58.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-34-58 patch sysinv 2020-05-23 19:34:58.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:39:58.295 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-39-58 patch sysinv 2020-05-23 19:39:58.295 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:44:58.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-44-58 patch sysinv 2020-05-23 19:44:58.303 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:49:58.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-49-58 patch sysinv 2020-05-23 19:49:58.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:54:58.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-54-58 patch sysinv 2020-05-23 19:54:58.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 19:59:58.439 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-19-59-58 patch sysinv 2020-05-23 19:59:58.440 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:04:58.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-04-58 patch sysinv 2020-05-23 20:04:58.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:09:58.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-09-58 patch sysinv 2020-05-23 20:09:58.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:14:58.510 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-14-58 patch sysinv 2020-05-23 20:14:58.510 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:19:58.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-19-58 patch sysinv 2020-05-23 20:19:58.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:24:58.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-24-58 patch sysinv 2020-05-23 20:24:58.570 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:29:58.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-29-58 patch sysinv 2020-05-23 20:29:58.581 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:34:59.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-34-59 patch sysinv 2020-05-23 20:34:59.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:39:58.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-39-58 patch sysinv 2020-05-23 20:39:58.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:44:58.665 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-44-58 patch sysinv 2020-05-23 20:44:58.665 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:49:58.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-49-58 patch sysinv 2020-05-23 20:49:58.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:54:58.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-54-58 patch sysinv 2020-05-23 20:54:58.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 20:59:58.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-20-59-58 patch sysinv 2020-05-23 20:59:58.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:04:58.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-04-58 patch sysinv 2020-05-23 21:04:58.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:09:58.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-09-58 patch sysinv 2020-05-23 21:09:58.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:14:58.915 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-14-58 patch sysinv 2020-05-23 21:14:58.916 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:19:58.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-19-58 patch sysinv 2020-05-23 21:19:58.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:24:58.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-24-58 patch sysinv 2020-05-23 21:24:58.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:29:59.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-29-58 patch sysinv 2020-05-23 21:29:59.006 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:34:59.343 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-34-59 patch sysinv 2020-05-23 21:34:59.343 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:39:58.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-39-58 patch sysinv 2020-05-23 21:39:58.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:44:58.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-44-58 patch sysinv 2020-05-23 21:44:58.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:49:59.062 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-49-59 patch sysinv 2020-05-23 21:49:59.062 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:54:59.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-54-59 patch sysinv 2020-05-23 21:54:59.152 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 21:59:59.214 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-21-59-59 patch sysinv 2020-05-23 21:59:59.214 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:04:59.252 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-04-59 patch sysinv 2020-05-23 22:04:59.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:09:59.186 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-09-59 patch sysinv 2020-05-23 22:09:59.187 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:14:59.283 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-14-59 patch sysinv 2020-05-23 22:14:59.284 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:19:59.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-19-59 patch sysinv 2020-05-23 22:19:59.262 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:24:59.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-24-59 patch sysinv 2020-05-23 22:24:59.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:29:59.303 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-29-59 patch sysinv 2020-05-23 22:29:59.304 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:34:59.423 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-34-59 patch sysinv 2020-05-23 22:34:59.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:39:59.357 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-39-59 patch sysinv 2020-05-23 22:39:59.357 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:44:59.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-44-59 patch sysinv 2020-05-23 22:44:59.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:49:59.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-49-59 patch sysinv 2020-05-23 22:49:59.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:54:59.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-54-59 patch sysinv 2020-05-23 22:54:59.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 22:59:59.518 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-22-59-59 patch sysinv 2020-05-23 22:59:59.518 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:04:59.617 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-04-59 patch sysinv 2020-05-23 23:04:59.618 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:09:59.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-09-59 patch sysinv 2020-05-23 23:09:59.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:14:59.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-14-59 patch sysinv 2020-05-23 23:14:59.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:19:59.668 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-19-59 patch sysinv 2020-05-23 23:19:59.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:24:59.766 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-24-59 patch sysinv 2020-05-23 23:24:59.766 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:29:59.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-29-59 patch sysinv 2020-05-23 23:29:59.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:34:59.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-34-59 patch sysinv 2020-05-23 23:34:59.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:39:59.760 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-39-59 patch sysinv 2020-05-23 23:39:59.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:44:59.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-44-59 patch sysinv 2020-05-23 23:44:59.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:49:59.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-49-59 patch sysinv 2020-05-23 23:49:59.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:54:59.856 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-54-59 patch sysinv 2020-05-23 23:54:59.856 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-23 23:59:59.916 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-23-23-59-59 patch sysinv 2020-05-23 23:59:59.916 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:04:59.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-04-59 patch sysinv 2020-05-24 00:04:59.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:09:59.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-09-59 patch sysinv 2020-05-24 00:09:59.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:14:59.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-14-59 patch sysinv 2020-05-24 00:14:59.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:20:00.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-19-59 patch sysinv 2020-05-24 00:20:00.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:24:59.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-24-59 patch sysinv 2020-05-24 00:24:59.970 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:30:00.208 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-30-00 patch sysinv 2020-05-24 00:30:00.209 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:35:00.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-34-59 patch sysinv 2020-05-24 00:35:00.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:40:00.047 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-40-00 patch sysinv 2020-05-24 00:40:00.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:45:00.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-45-00 patch sysinv 2020-05-24 00:45:00.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:50:00.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-50-00 patch sysinv 2020-05-24 00:50:00.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 00:55:00.235 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-00-55-00 patch sysinv 2020-05-24 00:55:00.236 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:00:00.211 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-00-00 patch sysinv 2020-05-24 01:00:00.211 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:05:00.284 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-05-00 patch sysinv 2020-05-24 01:05:00.284 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:10:00.291 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-10-00 patch sysinv 2020-05-24 01:10:00.292 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:15:00.462 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-15-00 patch sysinv 2020-05-24 01:15:00.463 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:20:00.356 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-20-00 patch sysinv 2020-05-24 01:20:00.357 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:25:00.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-25-00 patch sysinv 2020-05-24 01:25:00.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:30:00.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-30-00 patch sysinv 2020-05-24 01:30:00.415 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:35:00.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-35-00 patch sysinv 2020-05-24 01:35:00.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:40:00.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-40-00 patch sysinv 2020-05-24 01:40:00.469 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:45:00.493 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-45-00 patch sysinv 2020-05-24 01:45:00.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:50:00.618 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-50-00 patch sysinv 2020-05-24 01:50:00.618 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 01:55:00.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-01-55-00 patch sysinv 2020-05-24 01:55:00.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:00:00.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-00-00 patch sysinv 2020-05-24 02:00:00.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:05:00.605 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-05-00 patch sysinv 2020-05-24 02:05:00.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:10:00.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-10-00 patch sysinv 2020-05-24 02:10:00.710 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:15:00.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-15-00 patch sysinv 2020-05-24 02:15:00.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:20:00.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-20-00 patch sysinv 2020-05-24 02:20:00.710 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:25:00.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-25-00 patch sysinv 2020-05-24 02:25:00.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:30:00.800 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-30-00 patch sysinv 2020-05-24 02:30:00.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:35:00.859 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-35-00 patch sysinv 2020-05-24 02:35:00.860 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:40:00.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-40-00 patch sysinv 2020-05-24 02:40:00.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:45:00.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-45-00 patch sysinv 2020-05-24 02:45:00.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:50:00.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-50-00 patch sysinv 2020-05-24 02:50:00.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 02:55:01.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-02-55-00 patch sysinv 2020-05-24 02:55:01.019 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:00:01.061 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-00-01 patch sysinv 2020-05-24 03:00:01.062 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:05:00.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-05-00 patch sysinv 2020-05-24 03:05:00.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:10:01.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-10-01 patch sysinv 2020-05-24 03:10:01.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:15:00.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-15-00 patch sysinv 2020-05-24 03:15:00.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:20:01.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-20-00 patch sysinv 2020-05-24 03:20:01.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:25:01.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-25-01 patch sysinv 2020-05-24 03:25:01.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:30:01.128 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-30-01 patch sysinv 2020-05-24 03:30:01.128 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:35:01.229 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-35-01 patch sysinv 2020-05-24 03:35:01.229 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:40:01.246 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-40-01 patch sysinv 2020-05-24 03:40:01.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:45:01.342 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-45-01 patch sysinv 2020-05-24 03:45:01.343 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:50:01.388 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-50-01 patch sysinv 2020-05-24 03:50:01.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 03:55:01.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-03-55-01 patch sysinv 2020-05-24 03:55:01.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:00:01.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-00-01 patch sysinv 2020-05-24 04:00:01.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:05:01.388 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-05-01 patch sysinv 2020-05-24 04:05:01.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:10:01.413 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-10-01 patch sysinv 2020-05-24 04:10:01.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:15:01.508 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-15-01 patch sysinv 2020-05-24 04:15:01.510 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:20:01.499 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-20-01 patch sysinv 2020-05-24 04:20:01.500 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:25:01.489 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-25-01 patch sysinv 2020-05-24 04:25:01.489 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:30:01.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-30-01 patch sysinv 2020-05-24 04:30:01.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:35:01.979 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-35-01 patch sysinv 2020-05-24 04:35:01.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:40:01.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-40-01 patch sysinv 2020-05-24 04:40:01.640 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:45:01.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-45-01 patch sysinv 2020-05-24 04:45:01.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:50:01.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-50-01 patch sysinv 2020-05-24 04:50:01.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 04:55:01.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-04-55-01 patch sysinv 2020-05-24 04:55:01.652 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:00:01.671 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-00-01 patch sysinv 2020-05-24 05:00:01.672 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:05:01.722 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-05-01 patch sysinv 2020-05-24 05:05:01.723 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:10:01.705 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-10-01 patch sysinv 2020-05-24 05:10:01.705 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:15:01.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-15-01 patch sysinv 2020-05-24 05:15:01.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:20:01.872 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-20-01 patch sysinv 2020-05-24 05:20:01.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:25:01.858 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-25-01 patch sysinv 2020-05-24 05:25:01.859 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:30:01.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-30-01 patch sysinv 2020-05-24 05:30:01.943 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:35:02.452 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-35-02 patch sysinv 2020-05-24 05:35:02.452 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:40:01.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-40-01 patch sysinv 2020-05-24 05:40:01.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:45:01.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-45-01 patch sysinv 2020-05-24 05:45:01.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:50:02.020 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-50-02 patch sysinv 2020-05-24 05:50:02.021 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 05:55:02.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-05-55-01 patch sysinv 2020-05-24 05:55:02.015 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:00:02.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-00-02 patch sysinv 2020-05-24 06:00:02.177 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:05:02.107 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-05-02 patch sysinv 2020-05-24 06:05:02.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:10:02.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-10-02 patch sysinv 2020-05-24 06:10:02.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:15:02.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-15-02 patch sysinv 2020-05-24 06:15:02.518 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:20:02.232 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-20-02 patch sysinv 2020-05-24 06:20:02.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:25:02.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-25-02 patch sysinv 2020-05-24 06:25:02.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:30:02.265 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-30-02 patch sysinv 2020-05-24 06:30:02.266 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:35:02.644 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-35-02 patch sysinv 2020-05-24 06:35:02.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:40:02.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-40-02 patch sysinv 2020-05-24 06:40:02.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:45:02.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-45-02 patch sysinv 2020-05-24 06:45:02.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:50:02.399 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-50-02 patch sysinv 2020-05-24 06:50:02.400 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 06:55:02.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-06-55-02 patch sysinv 2020-05-24 06:55:02.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:00:02.496 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-00-02 patch sysinv 2020-05-24 07:00:02.497 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:05:02.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-05-02 patch sysinv 2020-05-24 07:05:02.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:10:02.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-10-02 patch sysinv 2020-05-24 07:10:02.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:15:02.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-15-02 patch sysinv 2020-05-24 07:15:02.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:20:02.635 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-20-02 patch sysinv 2020-05-24 07:20:02.635 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:25:02.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-25-02 patch sysinv 2020-05-24 07:25:02.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:30:02.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-30-02 patch sysinv 2020-05-24 07:30:02.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:35:02.660 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-35-02 patch sysinv 2020-05-24 07:35:02.660 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:40:02.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-40-02 patch sysinv 2020-05-24 07:40:02.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:45:02.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-45-02 patch sysinv 2020-05-24 07:45:02.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:50:02.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-50-02 patch sysinv 2020-05-24 07:50:02.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 07:55:02.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-07-55-02 patch sysinv 2020-05-24 07:55:02.763 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:00:02.820 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-00-02 patch sysinv 2020-05-24 08:00:02.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:05:02.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-05-02 patch sysinv 2020-05-24 08:05:02.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:10:02.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-10-02 patch sysinv 2020-05-24 08:10:02.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:15:02.945 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-15-02 patch sysinv 2020-05-24 08:15:02.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:20:03.089 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-20-03 patch sysinv 2020-05-24 08:20:03.089 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:25:03.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-25-02 patch sysinv 2020-05-24 08:25:03.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:30:02.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-30-02 patch sysinv 2020-05-24 08:30:02.975 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:35:03.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-35-03 patch sysinv 2020-05-24 08:35:03.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:40:03.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-40-02 patch sysinv 2020-05-24 08:40:03.219 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:45:03.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-45-03 patch sysinv 2020-05-24 08:45:03.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:50:03.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-50-03 patch sysinv 2020-05-24 08:50:03.076 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 08:55:03.205 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-08-55-03 patch sysinv 2020-05-24 08:55:03.205 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:00:03.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-00-03 patch sysinv 2020-05-24 09:00:03.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:05:03.249 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-05-03 patch sysinv 2020-05-24 09:05:03.250 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:10:03.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-10-03 patch sysinv 2020-05-24 09:10:03.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:15:03.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-15-03 patch sysinv 2020-05-24 09:15:03.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:20:03.300 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-20-03 patch sysinv 2020-05-24 09:20:03.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:25:03.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-25-03 patch sysinv 2020-05-24 09:25:03.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:30:03.470 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-30-03 patch sysinv 2020-05-24 09:30:03.470 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:35:03.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-35-03 patch sysinv 2020-05-24 09:35:03.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:40:03.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-40-03 patch sysinv 2020-05-24 09:40:03.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:45:03.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-45-03 patch sysinv 2020-05-24 09:45:03.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:50:03.444 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-50-03 patch sysinv 2020-05-24 09:50:03.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 09:55:03.553 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-09-55-03 patch sysinv 2020-05-24 09:55:03.553 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:00:03.486 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-00-03 patch sysinv 2020-05-24 10:00:03.487 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:05:03.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-05-03 patch sysinv 2020-05-24 10:05:03.545 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:10:03.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-10-03 patch sysinv 2020-05-24 10:10:03.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:15:03.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-15-03 patch sysinv 2020-05-24 10:15:03.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:20:03.746 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-20-03 patch sysinv 2020-05-24 10:20:03.746 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:25:03.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-25-03 patch sysinv 2020-05-24 10:25:03.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:30:03.689 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-30-03 patch sysinv 2020-05-24 10:30:03.690 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:35:04.167 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-35-04 patch sysinv 2020-05-24 10:35:04.168 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:40:03.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-40-03 patch sysinv 2020-05-24 10:40:03.686 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:45:03.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-45-03 patch sysinv 2020-05-24 10:45:03.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:50:03.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-50-03 patch sysinv 2020-05-24 10:50:03.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 10:55:03.870 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-10-55-03 patch sysinv 2020-05-24 10:55:03.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:00:03.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-00-03 patch sysinv 2020-05-24 11:00:03.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:05:03.912 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-05-03 patch sysinv 2020-05-24 11:05:03.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:10:04.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-10-04 patch sysinv 2020-05-24 11:10:04.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:15:03.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-15-03 patch sysinv 2020-05-24 11:15:03.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:20:04.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-20-03 patch sysinv 2020-05-24 11:20:04.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:25:04.011 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-25-03 patch sysinv 2020-05-24 11:25:04.012 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:30:04.067 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-30-04 patch sysinv 2020-05-24 11:30:04.067 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:35:04.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-35-03 patch sysinv 2020-05-24 11:35:04.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:40:04.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-40-04 patch sysinv 2020-05-24 11:40:04.048 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:45:04.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-45-04 patch sysinv 2020-05-24 11:45:04.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:50:04.223 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-50-04 patch sysinv 2020-05-24 11:50:04.224 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 11:55:04.246 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-11-55-04 patch sysinv 2020-05-24 11:55:04.246 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:00:04.279 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-00-04 patch sysinv 2020-05-24 12:00:04.279 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:05:04.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-05-04 patch sysinv 2020-05-24 12:05:04.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:10:04.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-10-04 patch sysinv 2020-05-24 12:10:04.353 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:15:04.391 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-15-04 patch sysinv 2020-05-24 12:15:04.391 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:20:04.436 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-20-04 patch sysinv 2020-05-24 12:20:04.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:25:04.329 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-25-04 patch sysinv 2020-05-24 12:25:04.330 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:30:04.458 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-30-04 patch sysinv 2020-05-24 12:30:04.459 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:35:04.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-35-04 patch sysinv 2020-05-24 12:35:04.434 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:40:04.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-40-04 patch sysinv 2020-05-24 12:40:04.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:45:04.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-45-04 patch sysinv 2020-05-24 12:45:04.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:50:04.508 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-50-04 patch sysinv 2020-05-24 12:50:04.509 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 12:55:04.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-12-55-04 patch sysinv 2020-05-24 12:55:04.620 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:00:04.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-00-04 patch sysinv 2020-05-24 13:00:04.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:05:04.568 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-05-04 patch sysinv 2020-05-24 13:05:04.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:10:04.581 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-10-04 patch sysinv 2020-05-24 13:10:04.582 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:15:04.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-15-04 patch sysinv 2020-05-24 13:15:04.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:20:04.765 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-20-04 patch sysinv 2020-05-24 13:20:04.765 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:25:04.703 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-25-04 patch sysinv 2020-05-24 13:25:04.704 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:30:04.789 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-30-04 patch sysinv 2020-05-24 13:30:04.790 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:35:04.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-35-04 patch sysinv 2020-05-24 13:35:04.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:40:05.201 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-40-05 patch sysinv 2020-05-24 13:40:05.201 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:45:05.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-45-04 patch sysinv 2020-05-24 13:45:05.106 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:50:04.879 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-50-04 patch sysinv 2020-05-24 13:50:04.879 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 13:55:05.129 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-13-55-05 patch sysinv 2020-05-24 13:55:05.130 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:00:04.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-00-04 patch sysinv 2020-05-24 14:00:04.914 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:05:05.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-05-05 patch sysinv 2020-05-24 14:05:05.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:10:04.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-10-04 patch sysinv 2020-05-24 14:10:04.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:15:04.997 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-15-04 patch sysinv 2020-05-24 14:15:04.997 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:20:05.088 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-20-05 patch sysinv 2020-05-24 14:20:05.088 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:25:05.130 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-25-05 patch sysinv 2020-05-24 14:25:05.130 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:30:05.219 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-30-05 patch sysinv 2020-05-24 14:30:05.220 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:35:05.182 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-35-05 patch sysinv 2020-05-24 14:35:05.182 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:40:05.765 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-40-05 patch sysinv 2020-05-24 14:40:05.766 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:45:05.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-45-05 patch sysinv 2020-05-24 14:45:05.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:50:05.268 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-50-05 patch sysinv 2020-05-24 14:50:05.268 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 14:55:05.353 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-14-55-05 patch sysinv 2020-05-24 14:55:05.353 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:00:05.296 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-00-05 patch sysinv 2020-05-24 15:00:05.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:05:05.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-05-05 patch sysinv 2020-05-24 15:05:05.317 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:10:05.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-10-05 patch sysinv 2020-05-24 15:10:05.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:15:05.472 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-15-05 patch sysinv 2020-05-24 15:15:05.472 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:20:05.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-20-05 patch sysinv 2020-05-24 15:20:05.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:25:05.579 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-25-05 patch sysinv 2020-05-24 15:25:05.580 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:30:05.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-30-05 patch sysinv 2020-05-24 15:30:05.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:35:05.573 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-35-05 patch sysinv 2020-05-24 15:35:05.573 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:40:05.587 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-40-05 patch sysinv 2020-05-24 15:40:05.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:45:05.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-45-05 patch sysinv 2020-05-24 15:45:05.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:50:05.665 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-50-05 patch sysinv 2020-05-24 15:50:05.666 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 15:55:05.656 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-15-55-05 patch sysinv 2020-05-24 15:55:05.657 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:00:05.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-00-05 patch sysinv 2020-05-24 16:00:05.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:05:05.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-05-05 patch sysinv 2020-05-24 16:05:05.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:10:05.882 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-10-05 patch sysinv 2020-05-24 16:10:05.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:15:05.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-15-05 patch sysinv 2020-05-24 16:15:05.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:20:05.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-20-05 patch sysinv 2020-05-24 16:20:05.877 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:25:05.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-25-05 patch sysinv 2020-05-24 16:25:05.826 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:30:05.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-30-05 patch sysinv 2020-05-24 16:30:05.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:35:05.882 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-35-05 patch sysinv 2020-05-24 16:35:05.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:40:05.881 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-40-05 patch sysinv 2020-05-24 16:40:05.881 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:45:05.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-45-05 patch sysinv 2020-05-24 16:45:05.914 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:50:05.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-50-05 patch sysinv 2020-05-24 16:50:05.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 16:55:06.183 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-16-55-06 patch sysinv 2020-05-24 16:55:06.183 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:00:05.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-00-05 patch sysinv 2020-05-24 17:00:05.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:05:06.068 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-05-06 patch sysinv 2020-05-24 17:05:06.069 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:10:06.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-10-06 patch sysinv 2020-05-24 17:10:06.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:15:06.181 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-15-06 patch sysinv 2020-05-24 17:15:06.181 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:20:06.226 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-20-06 patch sysinv 2020-05-24 17:20:06.227 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:25:06.186 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-25-06 patch sysinv 2020-05-24 17:25:06.187 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:30:06.279 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-30-06 patch sysinv 2020-05-24 17:30:06.280 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:35:06.325 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-35-06 patch sysinv 2020-05-24 17:35:06.326 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:40:06.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-40-06 patch sysinv 2020-05-24 17:40:06.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:45:06.323 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-45-06 patch sysinv 2020-05-24 17:45:06.324 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:50:06.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-50-06 patch sysinv 2020-05-24 17:50:06.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 17:55:06.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-17-55-06 patch sysinv 2020-05-24 17:55:06.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:00:06.387 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-00-06 patch sysinv 2020-05-24 18:00:06.388 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:05:06.492 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-05-06 patch sysinv 2020-05-24 18:05:06.493 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:10:06.447 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-10-06 patch sysinv 2020-05-24 18:10:06.448 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:15:06.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-15-06 patch sysinv 2020-05-24 18:15:06.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:20:06.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-20-06 patch sysinv 2020-05-24 18:20:06.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:25:06.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-25-06 patch sysinv 2020-05-24 18:25:06.676 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:30:06.579 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-30-06 patch sysinv 2020-05-24 18:30:06.579 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:35:06.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-35-06 patch sysinv 2020-05-24 18:35:06.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:40:07.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-40-07 patch sysinv 2020-05-24 18:40:07.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:45:06.671 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-45-06 patch sysinv 2020-05-24 18:45:06.671 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:50:06.703 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-50-06 patch sysinv 2020-05-24 18:50:06.704 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 18:55:06.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-18-55-06 patch sysinv 2020-05-24 18:55:06.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:00:06.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-00-06 patch sysinv 2020-05-24 19:00:06.763 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:05:06.737 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-05-06 patch sysinv 2020-05-24 19:05:06.737 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:10:06.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-10-06 patch sysinv 2020-05-24 19:10:06.863 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:15:07.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-15-07 patch sysinv 2020-05-24 19:15:07.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:20:06.830 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-20-06 patch sysinv 2020-05-24 19:20:06.831 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:25:06.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-25-06 patch sysinv 2020-05-24 19:25:06.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:30:06.979 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-30-06 patch sysinv 2020-05-24 19:30:06.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:35:07.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-35-07 patch sysinv 2020-05-24 19:35:07.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:40:06.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-40-06 patch sysinv 2020-05-24 19:40:06.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:45:07.179 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-45-07 patch sysinv 2020-05-24 19:45:07.180 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:50:07.013 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-50-06 patch sysinv 2020-05-24 19:50:07.013 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 19:55:07.107 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-19-55-07 patch sysinv 2020-05-24 19:55:07.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:00:07.197 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-00-07 patch sysinv 2020-05-24 20:00:07.197 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:05:07.220 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-05-07 patch sysinv 2020-05-24 20:05:07.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:10:07.232 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-10-07 patch sysinv 2020-05-24 20:10:07.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:15:07.323 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-15-07 patch sysinv 2020-05-24 20:15:07.324 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:20:07.288 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-20-07 patch sysinv 2020-05-24 20:20:07.288 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:25:07.280 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-25-07 patch sysinv 2020-05-24 20:25:07.281 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:30:07.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-30-07 patch sysinv 2020-05-24 20:30:07.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:35:07.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-35-07 patch sysinv 2020-05-24 20:35:07.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:40:07.403 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-40-07 patch sysinv 2020-05-24 20:40:07.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:45:07.400 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-45-07 patch sysinv 2020-05-24 20:45:07.401 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:50:07.489 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-50-07 patch sysinv 2020-05-24 20:50:07.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 20:55:07.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-20-55-07 patch sysinv 2020-05-24 20:55:07.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:00:07.696 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-00-07 patch sysinv 2020-05-24 21:00:07.697 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:05:07.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-05-07 patch sysinv 2020-05-24 21:05:07.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:10:07.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-10-07 patch sysinv 2020-05-24 21:10:07.568 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:15:07.697 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-15-07 patch sysinv 2020-05-24 21:15:07.698 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:20:07.633 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-20-07 patch sysinv 2020-05-24 21:20:07.636 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:25:07.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-25-07 patch sysinv 2020-05-24 21:25:07.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:30:07.763 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-30-07 patch sysinv 2020-05-24 21:30:07.764 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:35:07.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-35-07 patch sysinv 2020-05-24 21:35:07.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:40:07.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-40-07 patch sysinv 2020-05-24 21:40:07.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:45:07.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-45-07 patch sysinv 2020-05-24 21:45:07.822 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:50:07.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-50-07 patch sysinv 2020-05-24 21:50:07.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 21:55:07.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-21-55-07 patch sysinv 2020-05-24 21:55:07.969 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:00:07.887 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-00-07 patch sysinv 2020-05-24 22:00:07.888 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:05:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-05-07 patch sysinv 2020-05-24 22:05:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:10:07.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-10-07 patch sysinv 2020-05-24 22:10:07.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:15:08.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-15-08 patch sysinv 2020-05-24 22:15:08.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:20:07.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-20-07 patch sysinv 2020-05-24 22:20:07.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:25:07.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-25-07 patch sysinv 2020-05-24 22:25:07.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:30:08.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-30-08 patch sysinv 2020-05-24 22:30:08.053 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:35:08.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-35-07 patch sysinv 2020-05-24 22:35:08.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:40:08.059 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-40-08 patch sysinv 2020-05-24 22:40:08.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:45:08.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-45-08 patch sysinv 2020-05-24 22:45:08.076 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:50:08.177 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-50-08 patch sysinv 2020-05-24 22:50:08.177 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 22:55:08.232 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-22-55-08 patch sysinv 2020-05-24 22:55:08.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:00:08.258 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-00-08 patch sysinv 2020-05-24 23:00:08.258 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:05:08.238 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-05-08 patch sysinv 2020-05-24 23:05:08.243 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:10:08.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-10-08 patch sysinv 2020-05-24 23:10:08.329 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:15:08.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-15-08 patch sysinv 2020-05-24 23:15:08.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:20:08.439 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-20-08 patch sysinv 2020-05-24 23:20:08.440 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:25:08.353 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-25-08 patch sysinv 2020-05-24 23:25:08.354 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:30:08.455 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-30-08 patch sysinv 2020-05-24 23:30:08.455 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:35:08.511 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-35-08 patch sysinv 2020-05-24 23:35:08.512 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:40:08.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-40-08 patch sysinv 2020-05-24 23:40:08.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:45:08.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-45-08 patch sysinv 2020-05-24 23:45:08.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:50:08.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-50-08 patch sysinv 2020-05-24 23:50:08.592 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-24 23:55:08.749 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-24-23-55-08 patch sysinv 2020-05-24 23:55:08.750 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:00:08.796 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-00-08 patch sysinv 2020-05-25 00:00:08.797 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:05:08.593 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-05-08 patch sysinv 2020-05-25 00:05:08.593 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:10:08.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-10-08 patch sysinv 2020-05-25 00:10:08.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:15:08.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-15-08 patch sysinv 2020-05-25 00:15:08.731 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:20:08.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-20-08 patch sysinv 2020-05-25 00:20:08.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:25:08.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-25-08 patch sysinv 2020-05-25 00:25:08.802 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:30:08.818 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-30-08 patch sysinv 2020-05-25 00:30:08.819 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:35:08.864 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-35-08 patch sysinv 2020-05-25 00:35:08.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:40:08.773 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-40-08 patch sysinv 2020-05-25 00:40:08.773 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:45:08.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-45-08 patch sysinv 2020-05-25 00:45:08.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:50:08.893 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-50-08 patch sysinv 2020-05-25 00:50:08.894 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 00:55:08.958 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-00-55-08 patch sysinv 2020-05-25 00:55:08.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:00:08.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-00-08 patch sysinv 2020-05-25 01:00:08.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:05:09.058 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-05-09 patch sysinv 2020-05-25 01:05:09.058 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:10:08.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-10-08 patch sysinv 2020-05-25 01:10:08.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:15:08.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-15-08 patch sysinv 2020-05-25 01:15:08.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:20:08.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-20-08 patch sysinv 2020-05-25 01:20:08.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:25:09.071 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-25-09 patch sysinv 2020-05-25 01:25:09.071 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:30:09.412 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-30-09 patch sysinv 2020-05-25 01:30:09.412 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:35:09.152 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-35-09 patch sysinv 2020-05-25 01:35:09.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:40:09.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-40-09 patch sysinv 2020-05-25 01:40:09.164 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:45:09.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-45-09 patch sysinv 2020-05-25 01:45:09.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:50:09.397 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-50-09 patch sysinv 2020-05-25 01:50:09.397 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 01:55:09.350 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-01-55-09 patch sysinv 2020-05-25 01:55:09.350 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:00:09.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-00-09 patch sysinv 2020-05-25 02:00:09.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:05:09.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-05-09 patch sysinv 2020-05-25 02:05:09.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:10:09.398 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-10-09 patch sysinv 2020-05-25 02:10:09.398 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:15:09.471 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-15-09 patch sysinv 2020-05-25 02:15:09.471 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:20:09.459 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-20-09 patch sysinv 2020-05-25 02:20:09.459 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:25:09.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-25-09 patch sysinv 2020-05-25 02:25:09.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:30:09.501 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-30-09 patch sysinv 2020-05-25 02:30:09.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:35:09.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-35-09 patch sysinv 2020-05-25 02:35:09.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:40:09.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-40-09 patch sysinv 2020-05-25 02:40:09.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:45:10.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-45-10 patch sysinv 2020-05-25 02:45:10.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:50:09.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-50-09 patch sysinv 2020-05-25 02:50:09.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 02:55:09.570 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-02-55-09 patch sysinv 2020-05-25 02:55:09.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:00:09.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-00-09 patch sysinv 2020-05-25 03:00:09.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:05:09.750 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-05-09 patch sysinv 2020-05-25 03:05:09.751 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:10:09.725 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-10-09 patch sysinv 2020-05-25 03:10:09.725 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:15:09.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-15-09 patch sysinv 2020-05-25 03:15:09.807 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:20:09.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-20-09 patch sysinv 2020-05-25 03:20:09.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:25:09.846 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-25-09 patch sysinv 2020-05-25 03:25:09.846 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:30:09.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-30-09 patch sysinv 2020-05-25 03:30:09.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:35:10.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-35-09 patch sysinv 2020-05-25 03:35:10.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:40:09.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-40-09 patch sysinv 2020-05-25 03:40:09.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:45:10.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-45-10 patch sysinv 2020-05-25 03:45:10.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:50:09.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-50-09 patch sysinv 2020-05-25 03:50:09.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 03:55:09.944 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-03-55-09 patch sysinv 2020-05-25 03:55:09.944 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:00:09.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-00-09 patch sysinv 2020-05-25 04:00:09.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:05:10.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-05-10 patch sysinv 2020-05-25 04:05:10.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:10:10.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-10-10 patch sysinv 2020-05-25 04:10:10.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:15:10.227 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-15-10 patch sysinv 2020-05-25 04:15:10.228 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:20:10.258 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-20-10 patch sysinv 2020-05-25 04:20:10.259 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:25:10.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-25-10 patch sysinv 2020-05-25 04:25:10.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:30:10.217 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-30-10 patch sysinv 2020-05-25 04:30:10.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:35:10.337 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-35-10 patch sysinv 2020-05-25 04:35:10.337 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:40:10.285 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-40-10 patch sysinv 2020-05-25 04:40:10.286 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:45:10.244 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-45-10 patch sysinv 2020-05-25 04:45:10.244 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:50:10.407 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-50-10 patch sysinv 2020-05-25 04:50:10.408 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 04:55:10.360 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-04-55-10 patch sysinv 2020-05-25 04:55:10.360 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:00:10.446 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-00-10 patch sysinv 2020-05-25 05:00:10.447 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:05:10.402 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-05-10 patch sysinv 2020-05-25 05:05:10.403 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:10:10.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-10-10 patch sysinv 2020-05-25 05:10:10.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:15:10.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-15-10 patch sysinv 2020-05-25 05:15:10.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:20:10.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-20-10 patch sysinv 2020-05-25 05:20:10.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:25:10.562 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-25-10 patch sysinv 2020-05-25 05:25:10.563 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:30:10.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-30-10 patch sysinv 2020-05-25 05:30:10.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:35:10.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-35-10 patch sysinv 2020-05-25 05:35:10.638 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:40:10.716 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-40-10 patch sysinv 2020-05-25 05:40:10.716 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:45:10.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-45-10 patch sysinv 2020-05-25 05:45:10.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:50:10.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-50-10 patch sysinv 2020-05-25 05:50:10.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 05:55:10.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-05-55-10 patch sysinv 2020-05-25 05:55:10.778 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:00:11.041 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-00-10 patch sysinv 2020-05-25 06:00:11.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:05:10.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-05-10 patch sysinv 2020-05-25 06:05:10.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:10:10.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-10-10 patch sysinv 2020-05-25 06:10:10.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:15:10.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-15-10 patch sysinv 2020-05-25 06:15:10.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:20:10.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-20-10 patch sysinv 2020-05-25 06:20:10.885 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:25:11.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-25-10 patch sysinv 2020-05-25 06:25:11.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:30:11.090 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-30-11 patch sysinv 2020-05-25 06:30:11.091 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:35:10.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-35-10 patch sysinv 2020-05-25 06:35:10.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:40:11.086 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-40-11 patch sysinv 2020-05-25 06:40:11.087 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:45:11.642 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-45-11 patch sysinv 2020-05-25 06:45:11.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:50:11.015 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-50-10 patch sysinv 2020-05-25 06:50:11.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 06:55:11.080 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-06-55-11 patch sysinv 2020-05-25 06:55:11.080 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:00:11.119 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-00-11 patch sysinv 2020-05-25 07:00:11.120 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:05:11.206 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-05-11 patch sysinv 2020-05-25 07:05:11.207 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:10:11.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-10-11 patch sysinv 2020-05-25 07:10:11.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:15:11.262 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-15-11 patch sysinv 2020-05-25 07:15:11.262 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:20:11.241 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-20-11 patch sysinv 2020-05-25 07:20:11.241 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:25:11.320 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-25-11 patch sysinv 2020-05-25 07:25:11.320 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:30:11.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-30-11 patch sysinv 2020-05-25 07:30:11.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:35:11.428 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-35-11 patch sysinv 2020-05-25 07:35:11.428 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:40:11.500 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-40-11 patch sysinv 2020-05-25 07:40:11.502 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:45:11.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-45-11 patch sysinv 2020-05-25 07:45:11.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:50:11.423 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-50-11 patch sysinv 2020-05-25 07:50:11.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 07:55:11.542 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-07-55-11 patch sysinv 2020-05-25 07:55:11.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:00:11.606 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-00-11 patch sysinv 2020-05-25 08:00:11.607 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:05:11.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-05-11 patch sysinv 2020-05-25 08:05:11.539 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:10:11.610 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-10-11 patch sysinv 2020-05-25 08:10:11.610 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:15:11.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-15-11 patch sysinv 2020-05-25 08:15:11.670 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:20:11.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-20-11 patch sysinv 2020-05-25 08:20:11.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:25:11.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-25-11 patch sysinv 2020-05-25 08:25:11.599 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:30:11.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-30-11 patch sysinv 2020-05-25 08:30:11.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:35:11.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-35-11 patch sysinv 2020-05-25 08:35:11.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:40:11.805 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-40-11 patch sysinv 2020-05-25 08:40:11.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:45:11.721 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-45-11 patch sysinv 2020-05-25 08:45:11.722 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:50:11.813 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-50-11 patch sysinv 2020-05-25 08:50:11.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 08:55:11.927 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-08-55-11 patch sysinv 2020-05-25 08:55:11.927 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:00:11.921 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-00-11 patch sysinv 2020-05-25 09:00:11.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:05:11.842 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-05-11 patch sysinv 2020-05-25 09:05:11.842 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:10:12.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-10-12 patch sysinv 2020-05-25 09:10:12.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:15:11.902 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-15-11 patch sysinv 2020-05-25 09:15:11.902 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:20:11.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-20-11 patch sysinv 2020-05-25 09:20:11.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:25:12.039 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-25-12 patch sysinv 2020-05-25 09:25:12.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:30:12.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-30-11 patch sysinv 2020-05-25 09:30:12.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:35:12.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-35-12 patch sysinv 2020-05-25 09:35:12.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:40:12.188 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-40-12 patch sysinv 2020-05-25 09:40:12.189 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:45:12.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-45-12 patch sysinv 2020-05-25 09:45:12.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:50:12.195 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-50-12 patch sysinv 2020-05-25 09:50:12.196 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 09:55:12.296 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-09-55-12 patch sysinv 2020-05-25 09:55:12.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:00:12.254 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-00-12 patch sysinv 2020-05-25 10:00:12.255 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:05:12.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-05-12 patch sysinv 2020-05-25 10:05:12.431 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:10:12.380 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-10-12 patch sysinv 2020-05-25 10:10:12.380 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:15:12.415 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-15-12 patch sysinv 2020-05-25 10:15:12.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:20:12.392 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-20-12 patch sysinv 2020-05-25 10:20:12.392 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:25:12.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-25-12 patch sysinv 2020-05-25 10:25:12.389 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:30:12.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-30-12 patch sysinv 2020-05-25 10:30:12.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:35:12.456 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-35-12 patch sysinv 2020-05-25 10:35:12.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:40:12.480 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-40-12 patch sysinv 2020-05-25 10:40:12.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:45:12.480 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-45-12 patch sysinv 2020-05-25 10:45:12.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:50:12.513 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-50-12 patch sysinv 2020-05-25 10:50:12.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 10:55:12.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-10-55-12 patch sysinv 2020-05-25 10:55:12.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:00:12.661 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-00-12 patch sysinv 2020-05-25 11:00:12.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:05:12.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-05-12 patch sysinv 2020-05-25 11:05:12.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:10:12.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-10-12 patch sysinv 2020-05-25 11:10:12.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:15:12.740 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-15-12 patch sysinv 2020-05-25 11:15:12.740 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:20:12.733 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-20-12 patch sysinv 2020-05-25 11:20:12.734 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:25:12.768 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-25-12 patch sysinv 2020-05-25 11:25:12.768 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:30:12.786 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-30-12 patch sysinv 2020-05-25 11:30:12.786 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:35:12.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-35-12 patch sysinv 2020-05-25 11:35:12.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:40:12.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-40-12 patch sysinv 2020-05-25 11:40:12.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:45:12.815 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-45-12 patch sysinv 2020-05-25 11:45:12.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:50:12.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-50-12 patch sysinv 2020-05-25 11:50:12.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 11:55:12.959 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-11-55-12 patch sysinv 2020-05-25 11:55:12.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:00:13.196 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-00-13 patch sysinv 2020-05-25 12:00:13.196 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:05:12.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-05-12 patch sysinv 2020-05-25 12:05:12.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:10:13.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-10-12 patch sysinv 2020-05-25 12:10:13.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:15:13.113 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-15-13 patch sysinv 2020-05-25 12:15:13.113 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:20:13.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-20-13 patch sysinv 2020-05-25 12:20:13.025 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:25:13.247 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-25-13 patch sysinv 2020-05-25 12:25:13.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:30:13.141 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-30-13 patch sysinv 2020-05-25 12:30:13.141 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:35:13.190 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-35-13 patch sysinv 2020-05-25 12:35:13.190 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:40:13.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-40-13 patch sysinv 2020-05-25 12:40:13.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:45:13.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-45-13 patch sysinv 2020-05-25 12:45:13.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:50:13.318 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-50-13 patch sysinv 2020-05-25 12:50:13.318 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 12:55:13.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-12-55-13 patch sysinv 2020-05-25 12:55:13.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:00:13.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-00-13 patch sysinv 2020-05-25 13:00:13.363 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:05:13.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-05-13 patch sysinv 2020-05-25 13:05:13.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:10:13.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-10-13 patch sysinv 2020-05-25 13:10:13.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:15:13.436 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-15-13 patch sysinv 2020-05-25 13:15:13.436 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:20:13.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-20-13 patch sysinv 2020-05-25 13:20:13.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:25:13.518 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-25-13 patch sysinv 2020-05-25 13:25:13.519 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:30:13.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-30-13 patch sysinv 2020-05-25 13:30:13.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:35:13.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-35-13 patch sysinv 2020-05-25 13:35:13.561 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:40:13.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-40-13 patch sysinv 2020-05-25 13:40:13.543 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:45:13.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-45-13 patch sysinv 2020-05-25 13:45:13.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:50:13.665 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-50-13 patch sysinv 2020-05-25 13:50:13.666 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 13:55:13.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-13-55-13 patch sysinv 2020-05-25 13:55:13.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:00:13.686 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-00-13 patch sysinv 2020-05-25 14:00:13.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:05:13.894 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-05-13 patch sysinv 2020-05-25 14:05:13.894 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:10:13.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-10-13 patch sysinv 2020-05-25 14:10:13.744 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:15:13.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-15-13 patch sysinv 2020-05-25 14:15:13.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:20:13.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-20-13 patch sysinv 2020-05-25 14:20:13.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:25:13.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-25-13 patch sysinv 2020-05-25 14:25:13.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:30:13.889 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-30-13 patch sysinv 2020-05-25 14:30:13.890 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:35:13.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-35-13 patch sysinv 2020-05-25 14:35:13.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:40:13.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-40-13 patch sysinv 2020-05-25 14:40:13.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:45:13.938 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-45-13 patch sysinv 2020-05-25 14:45:13.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:50:14.107 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-50-14 patch sysinv 2020-05-25 14:50:14.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 14:55:13.966 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-14-55-13 patch sysinv 2020-05-25 14:55:13.966 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:00:14.006 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-00-13 patch sysinv 2020-05-25 15:00:14.007 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:05:14.020 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-05-13 patch sysinv 2020-05-25 15:05:14.020 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:10:14.061 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-10-14 patch sysinv 2020-05-25 15:10:14.063 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:15:14.199 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-15-14 patch sysinv 2020-05-25 15:15:14.200 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:20:14.184 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-20-14 patch sysinv 2020-05-25 15:20:14.185 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:25:14.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-25-14 patch sysinv 2020-05-25 15:25:14.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:30:14.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-30-14 patch sysinv 2020-05-25 15:30:14.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:35:14.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-35-14 patch sysinv 2020-05-25 15:35:14.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:40:14.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-40-14 patch sysinv 2020-05-25 15:40:14.310 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:45:14.760 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-45-14 patch sysinv 2020-05-25 15:45:14.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:50:14.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-50-14 patch sysinv 2020-05-25 15:50:14.387 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 15:55:14.422 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-15-55-14 patch sysinv 2020-05-25 15:55:14.423 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:00:14.471 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-00-14 patch sysinv 2020-05-25 16:00:14.472 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:05:14.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-05-14 patch sysinv 2020-05-25 16:05:14.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:10:14.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-10-14 patch sysinv 2020-05-25 16:10:14.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:15:14.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-15-14 patch sysinv 2020-05-25 16:15:14.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:20:14.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-20-14 patch sysinv 2020-05-25 16:20:14.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:25:14.551 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-25-14 patch sysinv 2020-05-25 16:25:14.551 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:30:14.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-30-14 patch sysinv 2020-05-25 16:30:14.763 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:35:14.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-35-14 patch sysinv 2020-05-25 16:35:14.663 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:40:14.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-40-14 patch sysinv 2020-05-25 16:40:14.758 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:45:15.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-45-14 patch sysinv 2020-05-25 16:45:15.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:50:14.894 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-50-14 patch sysinv 2020-05-25 16:50:14.895 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 16:55:14.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-16-55-14 patch sysinv 2020-05-25 16:55:14.789 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:00:15.004 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-00-14 patch sysinv 2020-05-25 17:00:15.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:05:14.803 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-05-14 patch sysinv 2020-05-25 17:05:14.804 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:10:14.825 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-10-14 patch sysinv 2020-05-25 17:10:14.826 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:15:14.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-15-14 patch sysinv 2020-05-25 17:15:14.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:20:14.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-20-14 patch sysinv 2020-05-25 17:20:14.929 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:25:15.075 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-25-15 patch sysinv 2020-05-25 17:25:15.076 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:30:14.914 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-30-14 patch sysinv 2020-05-25 17:30:14.914 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:35:14.957 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-35-14 patch sysinv 2020-05-25 17:35:14.958 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:40:14.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-40-14 patch sysinv 2020-05-25 17:40:14.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:45:15.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-45-14 patch sysinv 2020-05-25 17:45:15.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:50:15.167 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-50-15 patch sysinv 2020-05-25 17:50:15.168 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 17:55:15.171 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-17-55-15 patch sysinv 2020-05-25 17:55:15.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:00:15.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-00-15 patch sysinv 2020-05-25 18:00:15.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:05:15.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-05-15 patch sysinv 2020-05-25 18:05:15.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:10:15.252 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-10-15 patch sysinv 2020-05-25 18:10:15.253 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:15:15.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-15-15 patch sysinv 2020-05-25 18:15:15.355 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:20:15.382 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-20-15 patch sysinv 2020-05-25 18:20:15.383 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:25:15.266 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-25-15 patch sysinv 2020-05-25 18:25:15.267 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:30:15.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-30-15 patch sysinv 2020-05-25 18:30:15.294 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:35:15.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-35-15 patch sysinv 2020-05-25 18:35:15.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:40:15.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-40-15 patch sysinv 2020-05-25 18:40:15.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:45:15.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-45-15 patch sysinv 2020-05-25 18:45:15.404 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:50:15.505 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-50-15 patch sysinv 2020-05-25 18:50:15.506 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 18:55:15.493 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-18-55-15 patch sysinv 2020-05-25 18:55:15.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:00:15.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-00-15 patch sysinv 2020-05-25 19:00:15.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:05:15.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-05-15 patch sysinv 2020-05-25 19:05:15.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:10:15.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-10-15 patch sysinv 2020-05-25 19:10:15.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:15:15.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-15-15 patch sysinv 2020-05-25 19:15:15.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:20:15.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-20-15 patch sysinv 2020-05-25 19:20:15.893 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:25:15.696 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-25-15 patch sysinv 2020-05-25 19:25:15.697 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:30:15.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-30-15 patch sysinv 2020-05-25 19:30:15.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:35:15.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-35-15 patch sysinv 2020-05-25 19:35:15.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:40:15.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-40-15 patch sysinv 2020-05-25 19:40:15.744 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:45:15.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-45-15 patch sysinv 2020-05-25 19:45:15.923 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:50:15.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-50-15 patch sysinv 2020-05-25 19:50:15.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 19:55:16.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-19-55-16 patch sysinv 2020-05-25 19:55:16.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:00:15.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-00-15 patch sysinv 2020-05-25 20:00:15.816 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:05:15.971 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-05-15 patch sysinv 2020-05-25 20:05:15.972 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:10:16.050 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-10-16 patch sysinv 2020-05-25 20:10:16.051 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:15:15.923 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-15-15 patch sysinv 2020-05-25 20:15:15.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:20:15.944 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-20-15 patch sysinv 2020-05-25 20:20:15.945 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:25:16.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-25-15 patch sysinv 2020-05-25 20:25:16.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:30:16.134 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-30-16 patch sysinv 2020-05-25 20:30:16.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:35:16.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-35-16 patch sysinv 2020-05-25 20:35:16.105 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:40:16.112 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-40-16 patch sysinv 2020-05-25 20:40:16.112 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:45:16.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-45-16 patch sysinv 2020-05-25 20:45:16.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:50:16.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-50-16 patch sysinv 2020-05-25 20:50:16.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 20:55:16.292 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-20-55-16 patch sysinv 2020-05-25 20:55:16.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:00:16.430 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-00-16 patch sysinv 2020-05-25 21:00:16.431 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:05:16.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-05-16 patch sysinv 2020-05-25 21:05:16.249 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:10:16.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-10-16 patch sysinv 2020-05-25 21:10:16.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:15:16.478 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-15-16 patch sysinv 2020-05-25 21:15:16.479 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:20:16.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-20-16 patch sysinv 2020-05-25 21:20:16.378 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:25:16.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-25-16 patch sysinv 2020-05-25 21:25:16.336 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:30:16.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-30-16 patch sysinv 2020-05-25 21:30:16.491 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:35:16.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-35-16 patch sysinv 2020-05-25 21:35:16.570 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:40:16.526 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-40-16 patch sysinv 2020-05-25 21:40:16.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:45:16.869 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-45-16 patch sysinv 2020-05-25 21:45:16.869 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:50:16.622 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-50-16 patch sysinv 2020-05-25 21:50:16.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 21:55:16.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-21-55-16 patch sysinv 2020-05-25 21:55:16.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:00:16.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-00-16 patch sysinv 2020-05-25 22:00:16.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:05:16.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-05-16 patch sysinv 2020-05-25 22:05:16.715 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:10:16.744 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-10-16 patch sysinv 2020-05-25 22:10:16.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:15:16.760 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-15-16 patch sysinv 2020-05-25 22:15:16.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:20:16.843 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-20-16 patch sysinv 2020-05-25 22:20:16.847 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:25:16.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-25-16 patch sysinv 2020-05-25 22:25:16.729 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:30:16.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-30-16 patch sysinv 2020-05-25 22:30:16.769 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:35:16.895 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-35-16 patch sysinv 2020-05-25 22:35:16.895 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:40:16.882 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-40-16 patch sysinv 2020-05-25 22:40:16.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:45:17.265 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-45-17 patch sysinv 2020-05-25 22:45:17.266 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:50:16.847 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-50-16 patch sysinv 2020-05-25 22:50:16.847 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 22:55:16.891 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-22-55-16 patch sysinv 2020-05-25 22:55:16.892 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:00:16.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-00-16 patch sysinv 2020-05-25 23:00:16.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:05:17.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-05-17 patch sysinv 2020-05-25 23:05:17.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:10:16.961 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-10-16 patch sysinv 2020-05-25 23:10:16.962 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:15:16.991 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-15-16 patch sysinv 2020-05-25 23:15:16.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:20:17.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-20-17 patch sysinv 2020-05-25 23:20:17.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:25:17.269 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-25-17 patch sysinv 2020-05-25 23:25:17.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:30:17.158 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-30-17 patch sysinv 2020-05-25 23:30:17.158 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:35:17.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-35-17 patch sysinv 2020-05-25 23:35:17.433 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:40:17.230 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-40-17 patch sysinv 2020-05-25 23:40:17.231 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:45:17.275 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-45-17 patch sysinv 2020-05-25 23:45:17.275 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:50:17.311 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-50-17 patch sysinv 2020-05-25 23:50:17.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-25 23:55:17.376 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-25-23-55-17 patch sysinv 2020-05-25 23:55:17.376 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:00:17.492 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-00-17 patch sysinv 2020-05-26 00:00:17.493 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:05:17.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-05-17 patch sysinv 2020-05-26 00:05:17.418 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:10:17.614 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-10-17 patch sysinv 2020-05-26 00:10:17.616 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:15:17.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-15-17 patch sysinv 2020-05-26 00:15:17.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:20:17.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-20-17 patch sysinv 2020-05-26 00:20:17.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:25:17.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-25-17 patch sysinv 2020-05-26 00:25:17.486 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:30:17.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-30-17 patch sysinv 2020-05-26 00:30:17.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:35:17.780 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-35-17 patch sysinv 2020-05-26 00:35:17.780 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:40:17.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-40-17 patch sysinv 2020-05-26 00:40:17.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:45:17.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-45-17 patch sysinv 2020-05-26 00:45:17.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:50:17.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-50-17 patch sysinv 2020-05-26 00:50:17.708 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 00:55:17.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-00-55-17 patch sysinv 2020-05-26 00:55:17.782 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:00:17.779 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-00-17 patch sysinv 2020-05-26 01:00:17.780 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:05:17.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-05-17 patch sysinv 2020-05-26 01:05:17.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:10:17.807 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-10-17 patch sysinv 2020-05-26 01:10:17.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:15:17.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-15-17 patch sysinv 2020-05-26 01:15:17.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:20:17.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-20-17 patch sysinv 2020-05-26 01:20:17.867 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:25:17.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-25-17 patch sysinv 2020-05-26 01:25:17.777 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:30:17.817 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-30-17 patch sysinv 2020-05-26 01:30:17.818 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:35:17.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-35-17 patch sysinv 2020-05-26 01:35:17.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:40:17.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-40-17 patch sysinv 2020-05-26 01:40:17.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:45:18.412 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-45-18 patch sysinv 2020-05-26 01:45:18.412 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:50:17.938 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-50-17 patch sysinv 2020-05-26 01:50:17.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 01:55:18.114 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-01-55-18 patch sysinv 2020-05-26 01:55:18.114 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:00:17.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-00-17 patch sysinv 2020-05-26 02:00:17.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:05:18.112 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-05-18 patch sysinv 2020-05-26 02:05:18.113 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:10:18.121 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-10-18 patch sysinv 2020-05-26 02:10:18.121 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:15:18.282 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-15-18 patch sysinv 2020-05-26 02:15:18.283 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:20:18.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-20-18 patch sysinv 2020-05-26 02:20:18.173 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:25:18.200 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-25-18 patch sysinv 2020-05-26 02:25:18.200 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:30:18.370 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-30-18 patch sysinv 2020-05-26 02:30:18.372 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:35:18.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-35-18 patch sysinv 2020-05-26 02:35:18.382 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:40:18.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-40-18 patch sysinv 2020-05-26 02:40:18.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:45:18.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-45-18 patch sysinv 2020-05-26 02:45:18.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:50:18.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-50-18 patch sysinv 2020-05-26 02:50:18.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 02:55:18.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-02-55-18 patch sysinv 2020-05-26 02:55:18.477 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:00:18.440 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-00-18 patch sysinv 2020-05-26 03:00:18.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:05:18.456 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-05-18 patch sysinv 2020-05-26 03:05:18.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:10:18.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-10-18 patch sysinv 2020-05-26 03:10:18.541 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:15:18.585 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-15-18 patch sysinv 2020-05-26 03:15:18.586 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:20:18.693 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-20-18 patch sysinv 2020-05-26 03:20:18.693 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:25:18.592 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-25-18 patch sysinv 2020-05-26 03:25:18.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:30:18.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-30-18 patch sysinv 2020-05-26 03:30:18.612 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:35:18.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-35-18 patch sysinv 2020-05-26 03:35:18.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:40:18.680 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-40-18 patch sysinv 2020-05-26 03:40:18.681 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:45:18.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-45-18 patch sysinv 2020-05-26 03:45:18.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:50:18.805 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-50-18 patch sysinv 2020-05-26 03:50:18.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 03:55:18.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-03-55-18 patch sysinv 2020-05-26 03:55:18.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:00:18.854 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-00-18 patch sysinv 2020-05-26 04:00:18.855 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:05:18.763 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-05-18 patch sysinv 2020-05-26 04:05:18.764 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:10:18.907 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-10-18 patch sysinv 2020-05-26 04:10:18.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:15:18.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-15-18 patch sysinv 2020-05-26 04:15:18.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:20:18.988 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-20-18 patch sysinv 2020-05-26 04:20:18.988 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:25:18.875 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-25-18 patch sysinv 2020-05-26 04:25:18.875 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:30:18.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-30-18 patch sysinv 2020-05-26 04:30:18.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:35:19.020 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-35-19 patch sysinv 2020-05-26 04:35:19.021 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:40:19.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-40-19 patch sysinv 2020-05-26 04:40:19.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:45:19.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-45-18 patch sysinv 2020-05-26 04:45:19.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:50:19.041 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-50-19 patch sysinv 2020-05-26 04:50:19.042 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 04:55:19.198 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-04-55-19 patch sysinv 2020-05-26 04:55:19.198 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:00:19.259 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-00-19 patch sysinv 2020-05-26 05:00:19.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:05:19.288 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-05-19 patch sysinv 2020-05-26 05:05:19.289 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:10:19.314 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-10-19 patch sysinv 2020-05-26 05:10:19.314 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:15:19.256 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-15-19 patch sysinv 2020-05-26 05:15:19.257 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:20:19.442 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-20-19 patch sysinv 2020-05-26 05:20:19.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:25:19.337 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-25-19 patch sysinv 2020-05-26 05:25:19.338 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:30:19.325 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-30-19 patch sysinv 2020-05-26 05:30:19.326 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:35:19.497 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-35-19 patch sysinv 2020-05-26 05:35:19.497 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:40:19.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-40-19 patch sysinv 2020-05-26 05:40:19.369 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:45:19.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-45-19 patch sysinv 2020-05-26 05:45:19.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:50:19.440 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-50-19 patch sysinv 2020-05-26 05:50:19.441 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 05:55:19.497 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-05-55-19 patch sysinv 2020-05-26 05:55:19.498 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:00:19.605 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-00-19 patch sysinv 2020-05-26 06:00:19.605 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:05:19.575 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-05-19 patch sysinv 2020-05-26 06:05:19.576 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:10:19.654 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-10-19 patch sysinv 2020-05-26 06:10:19.654 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:15:19.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-15-19 patch sysinv 2020-05-26 06:15:19.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:20:19.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-20-19 patch sysinv 2020-05-26 06:20:19.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:25:19.677 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-25-19 patch sysinv 2020-05-26 06:25:19.677 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:30:20.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-30-19 patch sysinv 2020-05-26 06:30:20.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:35:19.740 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-35-19 patch sysinv 2020-05-26 06:35:19.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:40:19.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-40-19 patch sysinv 2020-05-26 06:40:19.794 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:45:20.169 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-45-20 patch sysinv 2020-05-26 06:45:20.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:50:19.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-50-19 patch sysinv 2020-05-26 06:50:19.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 06:55:19.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-06-55-19 patch sysinv 2020-05-26 06:55:19.784 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:00:19.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-00-19 patch sysinv 2020-05-26 07:00:19.813 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:05:19.988 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-05-19 patch sysinv 2020-05-26 07:05:19.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:10:19.943 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-10-19 patch sysinv 2020-05-26 07:10:19.944 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:15:20.070 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-15-20 patch sysinv 2020-05-26 07:15:20.071 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:20:19.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-20-19 patch sysinv 2020-05-26 07:20:19.928 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:25:20.177 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-25-20 patch sysinv 2020-05-26 07:25:20.178 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:30:19.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-30-19 patch sysinv 2020-05-26 07:30:19.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:35:20.025 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-35-19 patch sysinv 2020-05-26 07:35:20.026 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:40:20.057 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-40-20 patch sysinv 2020-05-26 07:40:20.058 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:45:20.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-45-20 patch sysinv 2020-05-26 07:45:20.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:50:20.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-50-20 patch sysinv 2020-05-26 07:50:20.202 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 07:55:20.286 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-07-55-20 patch sysinv 2020-05-26 07:55:20.286 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:00:20.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-00-20 patch sysinv 2020-05-26 08:00:20.306 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:05:20.212 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-05-20 patch sysinv 2020-05-26 08:05:20.212 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:10:20.358 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-10-20 patch sysinv 2020-05-26 08:10:20.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:15:20.337 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-15-20 patch sysinv 2020-05-26 08:15:20.338 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:20:20.563 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-20-20 patch sysinv 2020-05-26 08:20:20.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:25:20.330 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-25-20 patch sysinv 2020-05-26 08:25:20.331 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:30:20.435 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-30-20 patch sysinv 2020-05-26 08:30:20.436 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:35:20.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-35-20 patch sysinv 2020-05-26 08:35:20.545 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:40:20.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-40-20 patch sysinv 2020-05-26 08:40:20.538 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:45:20.480 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-45-20 patch sysinv 2020-05-26 08:45:20.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:50:20.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-50-20 patch sysinv 2020-05-26 08:50:20.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 08:55:20.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-08-55-20 patch sysinv 2020-05-26 08:55:20.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:00:20.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-00-20 patch sysinv 2020-05-26 09:00:20.640 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:05:20.614 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-05-20 patch sysinv 2020-05-26 09:05:20.614 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:10:20.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-10-20 patch sysinv 2020-05-26 09:10:20.968 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:15:20.864 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-15-20 patch sysinv 2020-05-26 09:15:20.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:20:20.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-20-20 patch sysinv 2020-05-26 09:20:20.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:25:20.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-25-20 patch sysinv 2020-05-26 09:25:20.783 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:30:20.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-30-20 patch sysinv 2020-05-26 09:30:20.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:35:20.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-35-20 patch sysinv 2020-05-26 09:35:20.814 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:40:20.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-40-20 patch sysinv 2020-05-26 09:40:20.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:45:20.851 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-45-20 patch sysinv 2020-05-26 09:45:20.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:50:21.045 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-50-20 patch sysinv 2020-05-26 09:50:21.046 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 09:55:21.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-09-55-20 patch sysinv 2020-05-26 09:55:21.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:00:21.278 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-00-21 patch sysinv 2020-05-26 10:00:21.279 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:05:21.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-05-21 patch sysinv 2020-05-26 10:05:21.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:10:20.966 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-10-20 patch sysinv 2020-05-26 10:10:20.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:15:20.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-15-20 patch sysinv 2020-05-26 10:15:20.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:20:21.087 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-20-21 patch sysinv 2020-05-26 10:20:21.087 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:25:21.191 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-25-21 patch sysinv 2020-05-26 10:25:21.191 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:30:21.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-30-21 patch sysinv 2020-05-26 10:30:21.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:35:21.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-35-21 patch sysinv 2020-05-26 10:35:21.352 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:40:21.268 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-40-21 patch sysinv 2020-05-26 10:40:21.269 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:45:21.774 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-45-21 patch sysinv 2020-05-26 10:45:21.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:50:21.287 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-50-21 patch sysinv 2020-05-26 10:50:21.287 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 10:55:21.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-10-55-21 patch sysinv 2020-05-26 10:55:21.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:00:21.447 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-00-21 patch sysinv 2020-05-26 11:00:21.448 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:05:21.490 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-05-21 patch sysinv 2020-05-26 11:05:21.492 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:10:21.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-10-21 patch sysinv 2020-05-26 11:10:21.547 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:15:21.552 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-15-21 patch sysinv 2020-05-26 11:15:21.553 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:20:21.642 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-20-21 patch sysinv 2020-05-26 11:20:21.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:25:21.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-25-21 patch sysinv 2020-05-26 11:25:21.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:30:21.780 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-30-21 patch sysinv 2020-05-26 11:30:21.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:35:21.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-35-21 patch sysinv 2020-05-26 11:35:21.643 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:40:21.681 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-40-21 patch sysinv 2020-05-26 11:40:21.682 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:45:21.603 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-45-21 patch sysinv 2020-05-26 11:45:21.603 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:50:21.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-50-21 patch sysinv 2020-05-26 11:50:21.781 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 11:55:21.850 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-11-55-21 patch sysinv 2020-05-26 11:55:21.851 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:00:21.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-00-21 patch sysinv 2020-05-26 12:00:21.794 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:05:21.735 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-05-21 patch sysinv 2020-05-26 12:05:21.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:10:21.879 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-10-21 patch sysinv 2020-05-26 12:10:21.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:15:21.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-15-21 patch sysinv 2020-05-26 12:15:21.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:20:21.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-20-21 patch sysinv 2020-05-26 12:20:21.995 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:25:21.851 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-25-21 patch sysinv 2020-05-26 12:25:21.852 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:30:21.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-30-21 patch sysinv 2020-05-26 12:30:21.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:35:22.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-35-22 patch sysinv 2020-05-26 12:35:22.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:40:22.026 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-40-22 patch sysinv 2020-05-26 12:40:22.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:45:21.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-45-21 patch sysinv 2020-05-26 12:45:21.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:50:21.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-50-21 patch sysinv 2020-05-26 12:50:21.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 12:55:22.116 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-12-55-22 patch sysinv 2020-05-26 12:55:22.117 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:00:22.311 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-00-22 patch sysinv 2020-05-26 13:00:22.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:05:22.181 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-05-22 patch sysinv 2020-05-26 13:05:22.181 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:10:22.186 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-10-22 patch sysinv 2020-05-26 13:10:22.187 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:15:22.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-15-22 patch sysinv 2020-05-26 13:15:22.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:20:22.348 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-20-22 patch sysinv 2020-05-26 13:20:22.351 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:25:22.400 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-25-22 patch sysinv 2020-05-26 13:25:22.401 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:30:22.336 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-30-22 patch sysinv 2020-05-26 13:30:22.336 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:35:22.462 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-35-22 patch sysinv 2020-05-26 13:35:22.464 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:40:22.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-40-22 patch sysinv 2020-05-26 13:40:22.482 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:45:22.405 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-45-22 patch sysinv 2020-05-26 13:45:22.405 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:50:22.474 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-50-22 patch sysinv 2020-05-26 13:50:22.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 13:55:22.604 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-13-55-22 patch sysinv 2020-05-26 13:55:22.605 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:00:22.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-00-22 patch sysinv 2020-05-26 14:00:22.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:05:22.649 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-05-22 patch sysinv 2020-05-26 14:05:22.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:10:22.752 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-10-22 patch sysinv 2020-05-26 14:10:22.753 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:15:22.634 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-15-22 patch sysinv 2020-05-26 14:15:22.635 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:20:22.586 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-20-22 patch sysinv 2020-05-26 14:20:22.587 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:25:22.771 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-25-22 patch sysinv 2020-05-26 14:25:22.772 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:30:22.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-30-22 patch sysinv 2020-05-26 14:30:22.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:35:22.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-35-22 patch sysinv 2020-05-26 14:35:22.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:40:22.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-40-22 patch sysinv 2020-05-26 14:40:22.721 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:45:23.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-45-23 patch sysinv 2020-05-26 14:45:23.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:50:22.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-50-22 patch sysinv 2020-05-26 14:50:22.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 14:55:22.754 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-14-55-22 patch sysinv 2020-05-26 14:55:22.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:00:22.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-00-22 patch sysinv 2020-05-26 15:00:22.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:05:22.870 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-05-22 patch sysinv 2020-05-26 15:05:22.871 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:10:23.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-10-22 patch sysinv 2020-05-26 15:10:23.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:15:23.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-15-22 patch sysinv 2020-05-26 15:15:23.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:20:23.088 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-20-23 patch sysinv 2020-05-26 15:20:23.089 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:25:22.946 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-25-22 patch sysinv 2020-05-26 15:25:22.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:30:23.188 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-30-23 patch sysinv 2020-05-26 15:30:23.189 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:35:23.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-35-22 patch sysinv 2020-05-26 15:35:23.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:40:23.039 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-40-23 patch sysinv 2020-05-26 15:40:23.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:45:23.062 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-45-23 patch sysinv 2020-05-26 15:45:23.063 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:50:23.203 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-50-23 patch sysinv 2020-05-26 15:50:23.203 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 15:55:23.306 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-15-55-23 patch sysinv 2020-05-26 15:55:23.306 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:00:23.259 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-00-23 patch sysinv 2020-05-26 16:00:23.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:05:23.395 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-05-23 patch sysinv 2020-05-26 16:05:23.396 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:10:23.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-10-23 patch sysinv 2020-05-26 16:10:23.335 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:15:23.422 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-15-23 patch sysinv 2020-05-26 16:15:23.422 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:20:23.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-20-23 patch sysinv 2020-05-26 16:20:23.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:25:23.525 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-25-23 patch sysinv 2020-05-26 16:25:23.526 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:30:23.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-30-23 patch sysinv 2020-05-26 16:30:23.739 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:35:23.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-35-23 patch sysinv 2020-05-26 16:35:23.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:40:23.685 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-40-23 patch sysinv 2020-05-26 16:40:23.686 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:45:23.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-45-23 patch sysinv 2020-05-26 16:45:23.537 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:50:23.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-50-23 patch sysinv 2020-05-26 16:50:23.649 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 16:55:23.604 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-16-55-23 patch sysinv 2020-05-26 16:55:23.604 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:00:23.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-00-23 patch sysinv 2020-05-26 17:00:23.731 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:05:23.857 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-05-23 patch sysinv 2020-05-26 17:05:23.857 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:10:23.797 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-10-23 patch sysinv 2020-05-26 17:10:23.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:15:23.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-15-23 patch sysinv 2020-05-26 17:15:23.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:20:23.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-20-23 patch sysinv 2020-05-26 17:20:23.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:25:23.874 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-25-23 patch sysinv 2020-05-26 17:25:23.875 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:30:23.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-30-23 patch sysinv 2020-05-26 17:30:23.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:35:23.965 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-35-23 patch sysinv 2020-05-26 17:35:23.965 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:40:23.835 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-40-23 patch sysinv 2020-05-26 17:40:23.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:45:23.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-45-23 patch sysinv 2020-05-26 17:45:23.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:50:24.043 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-50-23 patch sysinv 2020-05-26 17:50:24.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 17:55:23.986 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-17-55-23 patch sysinv 2020-05-26 17:55:23.987 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:00:24.054 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-00-24 patch sysinv 2020-05-26 18:00:24.054 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:05:24.159 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-05-24 patch sysinv 2020-05-26 18:05:24.160 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:10:24.017 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-10-23 patch sysinv 2020-05-26 18:10:24.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:15:24.004 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-15-23 patch sysinv 2020-05-26 18:15:24.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:20:24.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-20-23 patch sysinv 2020-05-26 18:20:24.041 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:25:24.084 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-25-24 patch sysinv 2020-05-26 18:25:24.085 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:30:24.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-30-24 patch sysinv 2020-05-26 18:30:24.171 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:35:24.233 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-35-24 patch sysinv 2020-05-26 18:35:24.234 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:40:24.292 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-40-24 patch sysinv 2020-05-26 18:40:24.293 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:45:24.208 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-45-24 patch sysinv 2020-05-26 18:45:24.209 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:50:24.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-50-24 patch sysinv 2020-05-26 18:50:24.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 18:55:24.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-18-55-24 patch sysinv 2020-05-26 18:55:24.443 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:00:24.657 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-00-24 patch sysinv 2020-05-26 19:00:24.658 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:05:24.326 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-05-24 patch sysinv 2020-05-26 19:05:24.327 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:10:24.467 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-10-24 patch sysinv 2020-05-26 19:10:24.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:15:24.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-15-24 patch sysinv 2020-05-26 19:15:24.569 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:20:24.480 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-20-24 patch sysinv 2020-05-26 19:20:24.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:25:24.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-25-24 patch sysinv 2020-05-26 19:25:24.425 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:30:24.461 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-30-24 patch sysinv 2020-05-26 19:30:24.462 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:35:24.548 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-35-24 patch sysinv 2020-05-26 19:35:24.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:40:24.667 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-40-24 patch sysinv 2020-05-26 19:40:24.667 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:45:25.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-45-24 patch sysinv 2020-05-26 19:45:25.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:50:24.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-50-24 patch sysinv 2020-05-26 19:50:24.813 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 19:55:24.724 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-19-55-24 patch sysinv 2020-05-26 19:55:24.725 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:00:24.729 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-00-24 patch sysinv 2020-05-26 20:00:24.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:05:24.722 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-05-24 patch sysinv 2020-05-26 20:05:24.722 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:10:24.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-10-24 patch sysinv 2020-05-26 20:10:24.730 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:15:24.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-15-24 patch sysinv 2020-05-26 20:15:24.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:20:24.776 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-20-24 patch sysinv 2020-05-26 20:20:24.776 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:25:24.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-25-24 patch sysinv 2020-05-26 20:25:24.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:30:24.948 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-30-24 patch sysinv 2020-05-26 20:30:24.949 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:35:25.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-35-24 patch sysinv 2020-05-26 20:35:25.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:40:25.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-40-25 patch sysinv 2020-05-26 20:40:25.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:45:25.413 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-45-25 patch sysinv 2020-05-26 20:45:25.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:50:24.943 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-50-24 patch sysinv 2020-05-26 20:50:24.943 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 20:55:24.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-20-55-24 patch sysinv 2020-05-26 20:55:24.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:00:25.006 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-00-24 patch sysinv 2020-05-26 21:00:25.007 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:05:25.189 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-05-25 patch sysinv 2020-05-26 21:05:25.190 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:10:25.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-10-25 patch sysinv 2020-05-26 21:10:25.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:15:25.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-15-25 patch sysinv 2020-05-26 21:15:25.276 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:20:25.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-20-25 patch sysinv 2020-05-26 21:20:25.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:25:25.228 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-25-25 patch sysinv 2020-05-26 21:25:25.228 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:30:25.358 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-30-25 patch sysinv 2020-05-26 21:30:25.359 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:35:25.304 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-35-25 patch sysinv 2020-05-26 21:35:25.305 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:40:25.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-40-25 patch sysinv 2020-05-26 21:40:25.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:45:25.511 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-45-25 patch sysinv 2020-05-26 21:45:25.511 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:50:25.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-50-25 patch sysinv 2020-05-26 21:50:25.415 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 21:55:25.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-21-55-25 patch sysinv 2020-05-26 21:55:25.611 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:00:25.422 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-00-25 patch sysinv 2020-05-26 22:00:25.423 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:05:25.399 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-05-25 patch sysinv 2020-05-26 22:05:25.400 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:10:25.621 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-10-25 patch sysinv 2020-05-26 22:10:25.621 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:15:25.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-15-25 patch sysinv 2020-05-26 22:15:25.715 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:20:25.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-20-25 patch sysinv 2020-05-26 22:20:25.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:25:25.515 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-25-25 patch sysinv 2020-05-26 22:25:25.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:30:25.539 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-30-25 patch sysinv 2020-05-26 22:30:25.540 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:35:25.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-35-25 patch sysinv 2020-05-26 22:35:25.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:40:25.693 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-40-25 patch sysinv 2020-05-26 22:40:25.693 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:45:26.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-45-26 patch sysinv 2020-05-26 22:45:26.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:50:25.657 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-50-25 patch sysinv 2020-05-26 22:50:25.657 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 22:55:25.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-22-55-25 patch sysinv 2020-05-26 22:55:25.812 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:00:25.868 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-00-25 patch sysinv 2020-05-26 23:00:25.869 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:05:25.842 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-05-25 patch sysinv 2020-05-26 23:05:25.843 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:10:26.086 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-10-26 patch sysinv 2020-05-26 23:10:26.088 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:15:25.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-15-25 patch sysinv 2020-05-26 23:15:25.809 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:20:25.947 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-20-25 patch sysinv 2020-05-26 23:20:25.948 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:25:25.923 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-25-25 patch sysinv 2020-05-26 23:25:25.923 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:30:26.128 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-30-26 patch sysinv 2020-05-26 23:30:26.128 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:35:25.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-35-25 patch sysinv 2020-05-26 23:35:25.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:40:26.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-40-26 patch sysinv 2020-05-26 23:40:26.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:45:26.080 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-45-26 patch sysinv 2020-05-26 23:45:26.080 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:50:26.221 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-50-26 patch sysinv 2020-05-26 23:50:26.222 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-26 23:55:26.127 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-26-23-55-26 patch sysinv 2020-05-26 23:55:26.127 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:00:26.192 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-00-26 patch sysinv 2020-05-27 00:00:26.193 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:05:26.259 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-05-26 patch sysinv 2020-05-27 00:05:26.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:10:26.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-10-26 patch sysinv 2020-05-27 00:10:26.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:15:26.551 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-15-26 patch sysinv 2020-05-27 00:15:26.552 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:20:26.324 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-20-26 patch sysinv 2020-05-27 00:20:26.325 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:25:26.373 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-25-26 patch sysinv 2020-05-27 00:25:26.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:30:26.417 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-30-26 patch sysinv 2020-05-27 00:30:26.418 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:35:26.419 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-35-26 patch sysinv 2020-05-27 00:35:26.419 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:40:26.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-40-26 patch sysinv 2020-05-27 00:40:26.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:45:26.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-45-26 patch sysinv 2020-05-27 00:45:26.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:50:26.621 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-50-26 patch sysinv 2020-05-27 00:50:26.621 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 00:55:26.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-00-55-26 patch sysinv 2020-05-27 00:55:26.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:00:26.725 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-00-26 patch sysinv 2020-05-27 01:00:26.726 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:05:26.501 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-05-26 patch sysinv 2020-05-27 01:05:26.501 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:10:26.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-10-26 patch sysinv 2020-05-27 01:10:26.640 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:15:26.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-15-26 patch sysinv 2020-05-27 01:15:26.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:20:26.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-20-26 patch sysinv 2020-05-27 01:20:26.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:25:26.615 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-25-26 patch sysinv 2020-05-27 01:25:26.615 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:30:26.792 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-30-26 patch sysinv 2020-05-27 01:30:26.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:35:26.760 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-35-26 patch sysinv 2020-05-27 01:35:26.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:40:26.754 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-40-26 patch sysinv 2020-05-27 01:40:26.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:45:26.765 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-45-26 patch sysinv 2020-05-27 01:45:26.765 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:50:26.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-50-26 patch sysinv 2020-05-27 01:50:26.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 01:55:26.941 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-01-55-26 patch sysinv 2020-05-27 01:55:26.942 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:00:26.916 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-00-26 patch sysinv 2020-05-27 02:00:26.917 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:05:27.050 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-05-27 patch sysinv 2020-05-27 02:05:27.050 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:10:26.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-10-26 patch sysinv 2020-05-27 02:10:26.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:15:27.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-15-27 patch sysinv 2020-05-27 02:15:27.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:20:26.980 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-20-26 patch sysinv 2020-05-27 02:20:26.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:25:27.056 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-25-27 patch sysinv 2020-05-27 02:25:27.056 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:30:27.183 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-30-27 patch sysinv 2020-05-27 02:30:27.184 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:35:27.139 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-35-27 patch sysinv 2020-05-27 02:35:27.140 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:40:27.129 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-40-27 patch sysinv 2020-05-27 02:40:27.130 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:45:27.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-45-27 patch sysinv 2020-05-27 02:45:27.710 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:50:27.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-50-27 patch sysinv 2020-05-27 02:50:27.271 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 02:55:27.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-02-55-27 patch sysinv 2020-05-27 02:55:27.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:00:27.400 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-00-27 patch sysinv 2020-05-27 03:00:27.401 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:05:27.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-05-27 patch sysinv 2020-05-27 03:05:27.297 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:10:27.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-10-27 patch sysinv 2020-05-27 03:10:27.374 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:15:27.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-15-27 patch sysinv 2020-05-27 03:15:27.393 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:20:27.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-20-27 patch sysinv 2020-05-27 03:20:27.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:25:27.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-25-27 patch sysinv 2020-05-27 03:25:27.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:30:27.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-30-27 patch sysinv 2020-05-27 03:30:27.652 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:35:27.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-35-27 patch sysinv 2020-05-27 03:35:27.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:40:27.708 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-40-27 patch sysinv 2020-05-27 03:40:27.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:45:27.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-45-27 patch sysinv 2020-05-27 03:45:27.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:50:27.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-50-27 patch sysinv 2020-05-27 03:50:27.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 03:55:27.759 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-03-55-27 patch sysinv 2020-05-27 03:55:27.760 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:00:27.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-00-27 patch sysinv 2020-05-27 04:00:27.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:05:27.872 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-05-27 patch sysinv 2020-05-27 04:05:27.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:10:27.775 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-10-27 patch sysinv 2020-05-27 04:10:27.776 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:15:27.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-15-27 patch sysinv 2020-05-27 04:15:27.808 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:20:27.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-20-27 patch sysinv 2020-05-27 04:20:27.967 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:25:27.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-25-27 patch sysinv 2020-05-27 04:25:27.807 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:30:27.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-30-27 patch sysinv 2020-05-27 04:30:27.838 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:35:27.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-35-27 patch sysinv 2020-05-27 04:35:27.935 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:40:27.900 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-40-27 patch sysinv 2020-05-27 04:40:27.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:45:27.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-45-27 patch sysinv 2020-05-27 04:45:27.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:50:28.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-50-28 patch sysinv 2020-05-27 04:50:28.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 04:55:27.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-04-55-27 patch sysinv 2020-05-27 04:55:27.940 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:00:27.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-00-27 patch sysinv 2020-05-27 05:00:27.964 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:05:27.999 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-05-27 patch sysinv 2020-05-27 05:05:28.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:10:28.134 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-10-28 patch sysinv 2020-05-27 05:10:28.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:15:28.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-15-28 patch sysinv 2020-05-27 05:15:28.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:20:28.181 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-20-28 patch sysinv 2020-05-27 05:20:28.182 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:25:28.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-25-28 patch sysinv 2020-05-27 05:25:28.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:30:28.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-30-28 patch sysinv 2020-05-27 05:30:28.290 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:35:28.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-35-28 patch sysinv 2020-05-27 05:35:28.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:40:28.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-40-28 patch sysinv 2020-05-27 05:40:28.329 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:45:28.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-45-28 patch sysinv 2020-05-27 05:45:28.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:50:28.286 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-50-28 patch sysinv 2020-05-27 05:50:28.286 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 05:55:28.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-05-55-28 patch sysinv 2020-05-27 05:55:28.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:00:28.439 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-00-28 patch sysinv 2020-05-27 06:00:28.440 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:05:28.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-05-28 patch sysinv 2020-05-27 06:05:28.455 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:10:28.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-10-28 patch sysinv 2020-05-27 06:10:28.544 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:15:28.593 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-15-28 patch sysinv 2020-05-27 06:15:28.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:20:28.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-20-28 patch sysinv 2020-05-27 06:20:28.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:25:28.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-25-28 patch sysinv 2020-05-27 06:25:28.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:30:28.639 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-30-28 patch sysinv 2020-05-27 06:30:28.640 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:35:28.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-35-28 patch sysinv 2020-05-27 06:35:28.663 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:40:28.691 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-40-28 patch sysinv 2020-05-27 06:40:28.692 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:45:28.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-45-28 patch sysinv 2020-05-27 06:45:28.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:50:28.719 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-50-28 patch sysinv 2020-05-27 06:50:28.719 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 06:55:28.717 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-06-55-28 patch sysinv 2020-05-27 06:55:28.718 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:00:28.830 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-00-28 patch sysinv 2020-05-27 07:00:28.830 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:05:28.865 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-05-28 patch sysinv 2020-05-27 07:05:28.866 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:10:28.872 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-10-28 patch sysinv 2020-05-27 07:10:28.872 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:15:28.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-15-28 patch sysinv 2020-05-27 07:15:28.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:20:29.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-20-28 patch sysinv 2020-05-27 07:20:29.002 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:25:28.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-25-28 patch sysinv 2020-05-27 07:25:28.926 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:30:29.050 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-30-29 patch sysinv 2020-05-27 07:30:29.051 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:35:28.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-35-28 patch sysinv 2020-05-27 07:35:28.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:40:29.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-40-28 patch sysinv 2020-05-27 07:40:29.004 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:45:29.346 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-45-29 patch sysinv 2020-05-27 07:45:29.347 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:50:28.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-50-28 patch sysinv 2020-05-27 07:50:28.992 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 07:55:29.043 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-07-55-29 patch sysinv 2020-05-27 07:55:29.043 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:00:29.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-00-29 patch sysinv 2020-05-27 08:00:29.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:05:29.175 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-05-29 patch sysinv 2020-05-27 08:05:29.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:10:29.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-10-29 patch sysinv 2020-05-27 08:10:29.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:15:29.271 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-15-29 patch sysinv 2020-05-27 08:15:29.271 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:20:29.242 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-20-29 patch sysinv 2020-05-27 08:20:29.243 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:25:29.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-25-29 patch sysinv 2020-05-27 08:25:29.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:30:29.265 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-30-29 patch sysinv 2020-05-27 08:30:29.265 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:35:29.456 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-35-29 patch sysinv 2020-05-27 08:35:29.456 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:40:29.367 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-40-29 patch sysinv 2020-05-27 08:40:29.367 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:45:29.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-45-29 patch sysinv 2020-05-27 08:45:29.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:50:29.475 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-50-29 patch sysinv 2020-05-27 08:50:29.476 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 08:55:29.480 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-08-55-29 patch sysinv 2020-05-27 08:55:29.481 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:00:29.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-00-29 patch sysinv 2020-05-27 09:00:29.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:05:29.454 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-05-29 patch sysinv 2020-05-27 09:05:29.455 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:10:29.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-10-29 patch sysinv 2020-05-27 09:10:29.494 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:15:29.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-15-29 patch sysinv 2020-05-27 09:15:29.626 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:20:29.683 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-20-29 patch sysinv 2020-05-27 09:20:29.684 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:25:29.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-25-29 patch sysinv 2020-05-27 09:25:29.707 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:30:29.800 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-30-29 patch sysinv 2020-05-27 09:30:29.801 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:35:29.770 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-35-29 patch sysinv 2020-05-27 09:35:29.770 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:40:29.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-40-29 patch sysinv 2020-05-27 09:40:29.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:45:29.806 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-45-29 patch sysinv 2020-05-27 09:45:29.807 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:50:29.810 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-50-29 patch sysinv 2020-05-27 09:50:29.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 09:55:29.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-09-55-29 patch sysinv 2020-05-27 09:55:29.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:00:29.905 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-00-29 patch sysinv 2020-05-27 10:00:29.913 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:05:29.883 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-05-29 patch sysinv 2020-05-27 10:05:29.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:10:29.975 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-10-29 patch sysinv 2020-05-27 10:10:29.976 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:15:30.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-15-30 patch sysinv 2020-05-27 10:15:30.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:20:29.924 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-20-29 patch sysinv 2020-05-27 10:20:29.925 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:25:30.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-25-30 patch sysinv 2020-05-27 10:25:30.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:30:29.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-30-29 patch sysinv 2020-05-27 10:30:29.996 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:35:30.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-35-29 patch sysinv 2020-05-27 10:35:30.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:40:30.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-40-30 patch sysinv 2020-05-27 10:40:30.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:45:30.214 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-45-30 patch sysinv 2020-05-27 10:45:30.214 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:50:30.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-50-30 patch sysinv 2020-05-27 10:50:30.163 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 10:55:30.320 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-10-55-30 patch sysinv 2020-05-27 10:55:30.321 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:00:30.246 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-00-30 patch sysinv 2020-05-27 11:00:30.246 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:05:30.318 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-05-30 patch sysinv 2020-05-27 11:05:30.319 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:10:30.386 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-10-30 patch sysinv 2020-05-27 11:10:30.386 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:15:30.457 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-15-30 patch sysinv 2020-05-27 11:15:30.462 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:20:30.419 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-20-30 patch sysinv 2020-05-27 11:20:30.419 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:25:30.331 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-25-30 patch sysinv 2020-05-27 11:25:30.332 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:30:30.572 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-30-30 patch sysinv 2020-05-27 11:30:30.573 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:35:30.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-35-30 patch sysinv 2020-05-27 11:35:30.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:40:30.504 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-40-30 patch sysinv 2020-05-27 11:40:30.504 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:45:30.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-45-30 patch sysinv 2020-05-27 11:45:30.468 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:50:30.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-50-30 patch sysinv 2020-05-27 11:50:30.589 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 11:55:30.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-11-55-30 patch sysinv 2020-05-27 11:55:30.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:00:30.634 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-00-30 patch sysinv 2020-05-27 12:00:30.635 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:05:30.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-05-30 patch sysinv 2020-05-27 12:05:30.625 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:10:30.708 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-10-30 patch sysinv 2020-05-27 12:10:30.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:15:30.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-15-30 patch sysinv 2020-05-27 12:15:30.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:20:30.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-20-30 patch sysinv 2020-05-27 12:20:30.786 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:25:30.753 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-25-30 patch sysinv 2020-05-27 12:25:30.753 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:30:30.799 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-30-30 patch sysinv 2020-05-27 12:30:30.799 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:35:30.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-35-30 patch sysinv 2020-05-27 12:35:30.836 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:40:30.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-40-30 patch sysinv 2020-05-27 12:40:30.974 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:45:31.360 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-45-31 patch sysinv 2020-05-27 12:45:31.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:50:30.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-50-30 patch sysinv 2020-05-27 12:50:30.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 12:55:31.116 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-12-55-30 patch sysinv 2020-05-27 12:55:31.117 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:00:31.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-00-30 patch sysinv 2020-05-27 13:00:31.044 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:05:31.140 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-05-31 patch sysinv 2020-05-27 13:05:31.141 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:10:30.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-10-30 patch sysinv 2020-05-27 13:10:30.960 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:15:30.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-15-30 patch sysinv 2020-05-27 13:15:30.993 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:20:31.167 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-20-31 patch sysinv 2020-05-27 13:20:31.168 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:25:31.225 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-25-31 patch sysinv 2020-05-27 13:25:31.226 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:30:31.312 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-30-31 patch sysinv 2020-05-27 13:30:31.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:35:31.367 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-35-31 patch sysinv 2020-05-27 13:35:31.368 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:40:31.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-40-31 patch sysinv 2020-05-27 13:40:31.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:45:31.715 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-45-31 patch sysinv 2020-05-27 13:45:31.715 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:50:31.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-50-31 patch sysinv 2020-05-27 13:50:31.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 13:55:31.444 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-13-55-31 patch sysinv 2020-05-27 13:55:31.445 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:00:31.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-00-31 patch sysinv 2020-05-27 14:00:31.486 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:05:31.419 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-05-31 patch sysinv 2020-05-27 14:05:31.420 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:10:31.741 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-10-31 patch sysinv 2020-05-27 14:10:31.742 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:15:31.513 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-15-31 patch sysinv 2020-05-27 14:15:31.514 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:20:31.436 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-20-31 patch sysinv 2020-05-27 14:20:31.437 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:25:31.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-25-31 patch sysinv 2020-05-27 14:25:31.414 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:30:31.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-30-31 patch sysinv 2020-05-27 14:30:31.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:35:31.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-35-31 patch sysinv 2020-05-27 14:35:31.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:40:31.745 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-40-31 patch sysinv 2020-05-27 14:40:31.747 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:45:31.562 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-45-31 patch sysinv 2020-05-27 14:45:31.562 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:49:14.450 90339 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 14:49:14.746 90339 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 14:49:14.819 90339 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 14:49:15.103 90339 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 14:49:15.105 90339 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 14:49:15.248 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:49:15.253 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:49:15.328 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:49:15.332 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:49:15.333 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 14:49:15.359 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:49:45.348 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:49:45.353 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:49:45.441 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:49:45.444 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:49:45.444 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 14:49:45.471 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:50:15.474 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:50:15.480 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:50:15.617 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:50:15.618 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:50:15.618 90339 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 14:50:15.757 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:50:15.763 90339 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 14:50:15.764 90339 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10467, in perform_app_apply 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode) 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp self._download_images(app) 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:50:15.764 90339 ERROR sysinv.openstack.common.rpc.amqp sysinv 2020-05-27 14:50:31.623 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-50-31 patch sysinv 2020-05-27 14:50:31.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:55:29.188 90339 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 14:55:30.171 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 14:55:30.190 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139757709609360'} sysinv 2020-05-27 14:55:30.203 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 14:55:30.212 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":21,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":63,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":472,"data_bytes":1657424627,"bytes_used":1746993152,"bytes_avail":1596718628864,"bytes_total":1598465622016,"write_bytes_sec":245914,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":73,"active_gid":614130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90220","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139757708930512'} sysinv 2020-05-27 14:55:30.795 90339 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 14:55:31.541 90339 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 14:55:31.656 90339 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 14:55:31.820 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-14-55-31 patch sysinv 2020-05-27 14:55:31.821 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 14:55:31.869 90339 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 14:55:32.176 90339 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 14:55:32.253 90339 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 14:55:32.656 90339 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 14:55:32.657 90339 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 14:55:32.724 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:55:32.726 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:55:32.766 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:55:32.767 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:55:32.768 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 14:55:32.793 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:56:02.790 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:56:02.793 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:56:02.833 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:56:02.833 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:56:02.834 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 14:56:02.864 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:56:32.836 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 14:56:32.839 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 14:56:32.886 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:56:32.887 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 14:56:32.887 90339 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 14:56:33.050 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 14:56:33.068 90339 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 14:56:33.069 90339 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 15:00:31.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-00-31 patch sysinv 2020-05-27 15:00:31.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:02:29.370 90339 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 15:02:29.401 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:02:29.414 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139757708907664'} sysinv 2020-05-27 15:02:29.426 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:02:29.435 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":21,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":63,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":472,"data_bytes":1657740019,"bytes_used":1747881984,"bytes_avail":1596717740032,"bytes_total":1598465622016,"write_bytes_sec":187230,"read_op_per_sec":0,"write_op_per_sec":33},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":73,"active_gid":614130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90220","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139757709462352'} sysinv 2020-05-27 15:02:29.855 90339 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 15:02:29.916 90339 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 15:02:29.927 90339 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 15:02:30.162 90339 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 15:02:30.460 90339 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 15:02:30.534 90339 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 15:02:31.024 90339 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 15:02:31.025 90339 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 15:02:31.129 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:02:31.132 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:02:31.192 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:02:31.193 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:02:31.193 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:02:31.211 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:03:01.216 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:03:01.218 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:03:01.267 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:03:01.268 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:03:01.268 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:03:01.309 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:03:31.303 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:03:31.305 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:03:31.365 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:03:31.372 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:03:31.372 90339 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 15:03:31.548 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:03:31.619 90339 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 15:03:31.619 90339 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 15:05:31.664 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-05-31 patch sysinv 2020-05-27 15:05:31.665 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:09:29.716 90339 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 15:09:29.761 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:09:29.777 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139757708850768'} sysinv 2020-05-27 15:09:29.787 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:09:29.799 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":21,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":63,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":472,"data_bytes":1658096371,"bytes_used":1746808832,"bytes_avail":1596718813184,"bytes_total":1598465622016,"write_bytes_sec":278710,"read_op_per_sec":0,"write_op_per_sec":56},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":73,"active_gid":614130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90220","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139757708839760'} sysinv 2020-05-27 15:09:30.335 90339 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 15:09:30.744 90339 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 15:09:30.871 90339 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 15:09:31.028 90339 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 15:09:31.330 90339 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 15:09:31.386 90339 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 15:09:31.830 90339 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 15:09:31.831 90339 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 15:09:31.945 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:09:31.948 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:09:32.012 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:09:32.013 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:09:32.013 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:09:32.092 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:10:02.037 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:10:02.046 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:10:02.117 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:10:02.118 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:10:02.119 90339 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:10:02.199 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:10:31.880 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-10-31 patch sysinv 2020-05-27 15:10:31.881 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:10:32.120 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:10:32.123 90339 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:10:32.184 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:10:32.185 90339 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:10:32.186 90339 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 15:10:32.335 90339 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:10:32.373 90339 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 15:10:32.374 90339 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 15:12:29.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:12:29.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-27 15:12:29.115 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.165 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:12:29.224 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.478 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.478 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:29.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-29 patch sysinv 2020-05-27 15:12:29.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:30.084 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:12:30.085 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:12:30.086 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:12:30.087 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:12:30.087 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:12:39.499 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-39 patch sysinv 2020-05-27 15:12:39.499 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:39.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-39 patch sysinv 2020-05-27 15:12:39.546 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:43.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-43 patch sysinv 2020-05-27 15:12:43.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:12:43.549 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-12-43 patch sysinv 2020-05-27 15:12:43.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-43 patch sysinv 2020-05-27 15:12:43.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:43.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-43 patch sysinv 2020-05-27 15:12:43.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:12:43.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-12-43 patch sysinv 2020-05-27 15:12:49.521 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-49 patch sysinv 2020-05-27 15:12:49.522 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:49.571 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-49 patch sysinv 2020-05-27 15:12:49.572 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:59.541 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-59 patch sysinv 2020-05-27 15:12:59.541 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:12:59.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-59 patch sysinv 2020-05-27 15:12:59.588 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:00.001 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-12-59 patch sysinv 2020-05-27 15:13:00.002 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:00.024 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-00 patch sysinv 2020-05-27 15:13:00.071 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-00 patch sysinv 2020-05-27 15:13:00.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:00.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-00 patch sysinv 2020-05-27 15:13:00.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:00.183 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-00 patch sysinv 2020-05-27 15:13:04.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-04 patch sysinv 2020-05-27 15:13:04.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:04.052 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-04 patch sysinv 2020-05-27 15:13:04.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-04 patch sysinv 2020-05-27 15:13:04.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:04.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-04 patch sysinv 2020-05-27 15:13:04.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:04.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-04 patch sysinv 2020-05-27 15:13:09.564 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-09 patch sysinv 2020-05-27 15:13:09.565 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:09.753 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-09 patch sysinv 2020-05-27 15:13:09.753 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:19.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:19.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:19.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:19.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.585 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.585 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:19.628 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:19.629 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:20.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-19 patch sysinv 2020-05-27 15:13:20.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:20.037 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-20 patch sysinv 2020-05-27 15:13:20.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-20 patch sysinv 2020-05-27 15:13:20.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:20.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-20 patch sysinv 2020-05-27 15:13:20.122 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:20.181 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-20 patch sysinv 2020-05-27 15:13:29.607 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-29 patch sysinv 2020-05-27 15:13:29.607 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:29.651 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-29 patch sysinv 2020-05-27 15:13:29.652 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:30.034 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:13:30.036 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:13:30.038 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:13:30.041 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:13:30.041 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:13:38.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-38 patch sysinv 2020-05-27 15:13:38.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:38.552 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-38 patch sysinv 2020-05-27 15:13:38.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-38 patch sysinv 2020-05-27 15:13:38.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:38.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-38 patch sysinv 2020-05-27 15:13:38.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:38.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-38 patch sysinv 2020-05-27 15:13:39.629 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-39 patch sysinv 2020-05-27 15:13:39.629 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:39.672 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-39 patch sysinv 2020-05-27 15:13:39.672 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:40.015 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-39 patch sysinv 2020-05-27 15:13:40.016 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:40.047 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-40 patch sysinv 2020-05-27 15:13:40.101 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-40 patch sysinv 2020-05-27 15:13:40.101 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:40.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-40 patch sysinv 2020-05-27 15:13:40.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:40.210 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-40 patch sysinv 2020-05-27 15:13:49.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-49 patch sysinv 2020-05-27 15:13:49.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:49.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-49 patch sysinv 2020-05-27 15:13:49.695 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:54.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-54 patch sysinv 2020-05-27 15:13:54.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:54.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-54 patch sysinv 2020-05-27 15:13:54.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-54 patch sysinv 2020-05-27 15:13:54.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:54.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-54 patch sysinv 2020-05-27 15:13:54.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:13:54.205 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-13-54 patch sysinv 2020-05-27 15:13:59.667 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-59 patch sysinv 2020-05-27 15:13:59.668 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:13:59.705 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-59 patch sysinv 2020-05-27 15:13:59.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-13-59 patch sysinv 2020-05-27 15:14:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:00.025 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-00 patch sysinv 2020-05-27 15:14:00.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-00 patch sysinv 2020-05-27 15:14:00.072 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:00.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-00 patch sysinv 2020-05-27 15:14:00.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:00.180 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-00 patch sysinv 2020-05-27 15:14:09.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-09 patch sysinv 2020-05-27 15:14:09.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:09.738 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-09 patch sysinv 2020-05-27 15:14:09.739 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:13.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-13 patch sysinv 2020-05-27 15:14:13.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:13.547 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-13 patch sysinv 2020-05-27 15:14:13.603 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-13 patch sysinv 2020-05-27 15:14:13.603 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:13.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-13 patch sysinv 2020-05-27 15:14:13.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:13.735 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-13 patch sysinv 2020-05-27 15:14:19.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-19 patch sysinv 2020-05-27 15:14:19.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:19.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-19 patch sysinv 2020-05-27 15:14:19.755 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:20.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-19 patch sysinv 2020-05-27 15:14:20.018 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:20.041 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-20 patch sysinv 2020-05-27 15:14:20.093 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-20 patch sysinv 2020-05-27 15:14:20.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:20.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-20 patch sysinv 2020-05-27 15:14:20.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:20.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-20 patch sysinv 2020-05-27 15:14:29.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:29.052 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:29.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:29.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.719 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.719 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:29.771 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-29 patch sysinv 2020-05-27 15:14:29.771 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:30.138 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:14:30.139 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:14:30.140 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:14:30.140 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:14:30.141 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:14:39.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-39 patch sysinv 2020-05-27 15:14:39.743 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:39.792 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-39 patch sysinv 2020-05-27 15:14:39.793 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:40.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-39 patch sysinv 2020-05-27 15:14:40.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:40.029 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-40 patch sysinv 2020-05-27 15:14:40.073 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-40 patch sysinv 2020-05-27 15:14:40.073 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:40.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-40 patch sysinv 2020-05-27 15:14:40.123 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:40.179 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-40 patch sysinv 2020-05-27 15:14:48.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-48 patch sysinv 2020-05-27 15:14:48.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:48.054 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-48 patch sysinv 2020-05-27 15:14:48.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-48 patch sysinv 2020-05-27 15:14:48.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:48.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-48 patch sysinv 2020-05-27 15:14:48.157 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:14:48.244 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-14-48 patch sysinv 2020-05-27 15:14:49.768 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-49 patch sysinv 2020-05-27 15:14:49.768 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:49.953 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-49 patch sysinv 2020-05-27 15:14:49.954 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:59.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-59 patch sysinv 2020-05-27 15:14:59.785 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:14:59.833 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-59 patch sysinv 2020-05-27 15:14:59.833 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-14-59 patch sysinv 2020-05-27 15:15:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:00.024 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-00 patch sysinv 2020-05-27 15:15:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-00 patch sysinv 2020-05-27 15:15:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:00.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-00 patch sysinv 2020-05-27 15:15:00.127 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:00.183 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-00 patch sysinv 2020-05-27 15:15:04.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-04 patch sysinv 2020-05-27 15:15:04.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:04.551 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-04 patch sysinv 2020-05-27 15:15:04.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-04 patch sysinv 2020-05-27 15:15:04.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:04.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-04 patch sysinv 2020-05-27 15:15:04.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:04.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-04 patch sysinv 2020-05-27 15:15:09.810 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-09 patch sysinv 2020-05-27 15:15:09.811 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:09.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-09 patch sysinv 2020-05-27 15:15:09.853 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:19.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-19 patch sysinv 2020-05-27 15:15:19.827 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:19.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-19 patch sysinv 2020-05-27 15:15:19.876 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:20.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-19 patch sysinv 2020-05-27 15:15:20.022 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:20.045 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-20 patch sysinv 2020-05-27 15:15:20.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-20 patch sysinv 2020-05-27 15:15:20.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:20.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-20 patch sysinv 2020-05-27 15:15:20.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:20.215 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-20 patch sysinv 2020-05-27 15:15:23.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-23 patch sysinv 2020-05-27 15:15:23.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:23.547 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-23 patch sysinv 2020-05-27 15:15:23.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-23 patch sysinv 2020-05-27 15:15:23.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:23.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-23 patch sysinv 2020-05-27 15:15:23.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:23.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-23 patch sysinv 2020-05-27 15:15:29.849 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-29 patch sysinv 2020-05-27 15:15:29.850 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:29.901 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-29 patch sysinv 2020-05-27 15:15:29.902 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:30.216 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:15:30.217 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:15:30.217 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:15:30.218 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:15:30.219 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:15:38.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-38 patch sysinv 2020-05-27 15:15:38.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:38.611 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-38 patch sysinv 2020-05-27 15:15:38.655 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-38 patch sysinv 2020-05-27 15:15:38.656 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:38.705 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-38 patch sysinv 2020-05-27 15:15:38.706 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:38.761 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-38 patch sysinv 2020-05-27 15:15:39.872 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-39 patch sysinv 2020-05-27 15:15:39.873 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:39.914 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-39 patch sysinv 2020-05-27 15:15:39.915 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:40.008 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-39 patch sysinv 2020-05-27 15:15:40.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:40.027 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-40 patch sysinv 2020-05-27 15:15:40.078 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-40 patch sysinv 2020-05-27 15:15:40.078 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:40.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-40 patch sysinv 2020-05-27 15:15:40.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:40.181 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-40 patch sysinv 2020-05-27 15:15:49.886 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-49 patch sysinv 2020-05-27 15:15:49.887 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:49.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-49 patch sysinv 2020-05-27 15:15:49.939 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:58.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-58 patch sysinv 2020-05-27 15:15:58.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:58.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-58 patch sysinv 2020-05-27 15:15:58.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-58 patch sysinv 2020-05-27 15:15:58.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:58.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-58 patch sysinv 2020-05-27 15:15:58.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:15:58.705 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-15-58 patch sysinv 2020-05-27 15:15:59.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-59 patch sysinv 2020-05-27 15:15:59.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:15:59.956 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-59 patch sysinv 2020-05-27 15:15:59.956 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-15-59 patch sysinv 2020-05-27 15:16:00.003 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:00.024 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-00 patch sysinv 2020-05-27 15:16:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-00 patch sysinv 2020-05-27 15:16:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:00.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-00 patch sysinv 2020-05-27 15:16:00.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:00.184 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-00 patch sysinv 2020-05-27 15:16:09.931 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-09 patch sysinv 2020-05-27 15:16:09.932 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:09.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-09 patch sysinv 2020-05-27 15:16:09.985 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:14.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-14 patch sysinv 2020-05-27 15:16:14.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:14.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-14 patch sysinv 2020-05-27 15:16:14.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-14 patch sysinv 2020-05-27 15:16:14.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:14.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-14 patch sysinv 2020-05-27 15:16:14.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:14.205 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-14 patch sysinv 2020-05-27 15:16:19.952 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-19 patch sysinv 2020-05-27 15:16:19.952 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:20.012 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-19 patch sysinv 2020-05-27 15:16:20.012 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:20.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-20 patch sysinv 2020-05-27 15:16:20.050 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:20.070 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-20 patch sysinv 2020-05-27 15:16:20.119 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-20 patch sysinv 2020-05-27 15:16:20.120 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:20.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-20 patch sysinv 2020-05-27 15:16:20.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:20.230 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-20 patch sysinv 2020-05-27 15:16:29.972 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-29 patch sysinv 2020-05-27 15:16:29.973 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:30.023 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-30 patch sysinv 2020-05-27 15:16:30.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:30.026 90339 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 15:16:30.044 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:16:30.052 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139757709113552'} sysinv 2020-05-27 15:16:30.057 90339 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:16:30.065 90339 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":21,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":63,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":473,"data_bytes":1658223347,"bytes_used":1745735680,"bytes_avail":1596719886336,"bytes_total":1598465622016},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":73,"active_gid":614130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90220","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139757708665680'} sysinv 2020-05-27 15:16:30.406 90339 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 15:16:30.436 90339 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 15:16:30.446 90339 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 15:16:30.649 90339 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 15:16:30.950 90339 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 15:16:30.979 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:30.983 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:30.984 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:30.985 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_secret: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1947, in perform_app_apply 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app self._create_local_registry_secrets(app.name) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1002, in _create_local_registry_secrets 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app self._kube.kube_get_secret(DOCKER_REGISTRY_SECRET, ns)): 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 194, in kube_get_secret 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app c.read_namespaced_secret(name, namespace) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 19486, in read_namespaced_secret 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app (data) = self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 19577, in read_namespaced_secret_with_http_info 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app collection_formats=collection_formats) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app _request_timeout=_request_timeout) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app headers=headers) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app query_params=query_params) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app headers=headers) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app **urlopen_kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app return self.urlopen(method, url, **extra_kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app _stacktrace=sys.exc_info()[2]) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:16:30.986 90339 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (psycopg2.DataError) value too long for type character varying(255) [SQL: 'UPDATE kube_app SET updated_at=%(updated_at)s, status=%(status)s, progress=%(progress)s WHERE kube_app.id = %(id_1)s'] [parameters: {'status': u'apply-failed', 'progress': u"HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Ca ... (16 characters truncated) ... ctionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',))", 'id_1': 1, 'updated_at': datetime.datetime(2020, 5, 27, 15, 16, 31, 135283)}]: DataError: value too long for type character varying(255) 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters context) 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters DataError: value too long for type character varying(255) 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters 2020-05-27 15:16:31.136 90339 ERROR oslo_db.sqlalchemy.exc_filters sysinv 2020-05-27 15:16:31.182 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:31.191 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:31.196 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:16:31.197 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:16:31.197 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:16:33.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-33 patch sysinv 2020-05-27 15:16:33.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:33.549 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-33 patch sysinv 2020-05-27 15:16:33.602 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-33 patch sysinv 2020-05-27 15:16:33.602 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:33.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-33 patch sysinv 2020-05-27 15:16:33.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:33.707 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-33 patch sysinv 2020-05-27 15:16:40.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-39 patch sysinv 2020-05-27 15:16:40.000 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:40.055 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:40.056 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:40.107 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:40.108 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:40.131 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:40.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:40.176 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:40.227 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:40.228 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:40.425 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-40 patch sysinv 2020-05-27 15:16:48.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-48 patch sysinv 2020-05-27 15:16:48.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:48.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-48 patch sysinv 2020-05-27 15:16:48.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-48 patch sysinv 2020-05-27 15:16:48.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:48.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-48 patch sysinv 2020-05-27 15:16:48.649 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:16:48.705 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-16-48 patch sysinv 2020-05-27 15:16:50.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-49 patch sysinv 2020-05-27 15:16:50.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:16:50.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-50 patch sysinv 2020-05-27 15:16:50.060 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:00.009 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-16-59 patch sysinv 2020-05-27 15:17:00.010 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:00.032 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:00.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:00.181 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.248 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:00.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-00 patch sysinv 2020-05-27 15:17:00.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:07.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-07 patch sysinv 2020-05-27 15:17:07.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:07.551 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-07 patch sysinv 2020-05-27 15:17:07.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-07 patch sysinv 2020-05-27 15:17:07.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:07.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-07 patch sysinv 2020-05-27 15:17:07.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:07.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-07 patch sysinv 2020-05-27 15:17:10.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-10 patch sysinv 2020-05-27 15:17:10.049 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:10.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-10 patch sysinv 2020-05-27 15:17:10.100 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:20.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-19 patch sysinv 2020-05-27 15:17:20.024 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:20.044 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:20.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:20.214 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.291 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.291 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:20.339 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-20 patch sysinv 2020-05-27 15:17:20.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:23.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-23 patch sysinv 2020-05-27 15:17:23.537 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:23.566 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-23 patch sysinv 2020-05-27 15:17:23.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-23 patch sysinv 2020-05-27 15:17:23.624 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:23.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-23 patch sysinv 2020-05-27 15:17:23.669 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:23.676 90934 WARNING wsme.api [-] Client-side error: Application-apply rejected: install/update is already in progress.: ClientSideError: Application-apply rejected: install/update is already in progress. sysinv 2020-05-27 15:17:23.740 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-23 patch sysinv 2020-05-27 15:17:30.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-30 patch sysinv 2020-05-27 15:17:30.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:30.137 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-30 patch sysinv 2020-05-27 15:17:30.138 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:40.014 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-39 patch sysinv 2020-05-27 15:17:40.015 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:40.040 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.079 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.079 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:40.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.131 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:40.193 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:40.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-40 patch sysinv 2020-05-27 15:17:40.328 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:42.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-42 patch sysinv 2020-05-27 15:17:42.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:42.054 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-42 patch sysinv 2020-05-27 15:17:42.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-42 patch sysinv 2020-05-27 15:17:42.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:42.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-42 patch sysinv 2020-05-27 15:17:42.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:42.209 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-42 patch sysinv 2020-05-27 15:17:50.124 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-50 patch sysinv 2020-05-27 15:17:50.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:50.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-50 patch sysinv 2020-05-27 15:17:50.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:58.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-58 patch sysinv 2020-05-27 15:17:58.035 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:58.054 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-58 patch sysinv 2020-05-27 15:17:58.101 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-58 patch sysinv 2020-05-27 15:17:58.102 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:17:58.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-17-58 patch sysinv 2020-05-27 15:17:58.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:17:58.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-17-58 patch sysinv 2020-05-27 15:18:00.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:00.053 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:00.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:00.349 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.429 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:00.484 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-00 patch sysinv 2020-05-27 15:18:00.485 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:10.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-10 patch sysinv 2020-05-27 15:18:10.170 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:10.208 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-10 patch sysinv 2020-05-27 15:18:10.209 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:17.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-17 patch sysinv 2020-05-27 15:18:17.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:17.048 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-17 patch sysinv 2020-05-27 15:18:17.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-17 patch sysinv 2020-05-27 15:18:17.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:17.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-17 patch sysinv 2020-05-27 15:18:17.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:17.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-17 patch sysinv 2020-05-27 15:18:20.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:20.054 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:20.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:20.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.270 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:20.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-20 patch sysinv 2020-05-27 15:18:20.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:30.217 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-30 patch sysinv 2020-05-27 15:18:30.218 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:30.267 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-30 patch sysinv 2020-05-27 15:18:30.267 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:30.357 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:18:30.359 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:18:30.360 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:18:30.363 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:18:30.364 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:18:33.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-33 patch sysinv 2020-05-27 15:18:33.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:33.053 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-33 patch sysinv 2020-05-27 15:18:33.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-33 patch sysinv 2020-05-27 15:18:33.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:33.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-33 patch sysinv 2020-05-27 15:18:33.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:33.213 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-33 patch sysinv 2020-05-27 15:18:40.039 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:40.065 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.118 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:40.179 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.179 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:40.237 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.313 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.314 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:40.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-40 patch sysinv 2020-05-27 15:18:40.362 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:50.234 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-50 patch sysinv 2020-05-27 15:18:50.234 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:50.285 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-50 patch sysinv 2020-05-27 15:18:50.285 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:51.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-51 patch sysinv 2020-05-27 15:18:51.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:51.556 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-51 patch sysinv 2020-05-27 15:18:51.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-51 patch sysinv 2020-05-27 15:18:51.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:18:51.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-18-51 patch sysinv 2020-05-27 15:18:51.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:18:51.715 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-18-51 patch sysinv 2020-05-27 15:19:00.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.036 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:00.055 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:00.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:00.204 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.273 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:00.321 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-00 patch sysinv 2020-05-27 15:19:00.321 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:07.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-07 patch sysinv 2020-05-27 15:19:07.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:07.549 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-07 patch sysinv 2020-05-27 15:19:07.601 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-07 patch sysinv 2020-05-27 15:19:07.602 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:07.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-07 patch sysinv 2020-05-27 15:19:07.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:07.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-07 patch sysinv 2020-05-27 15:19:10.278 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-10 patch sysinv 2020-05-27 15:19:10.278 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:10.326 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-10 patch sysinv 2020-05-27 15:19:10.326 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:20.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.299 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:20.344 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.344 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:20.524 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.525 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:20.545 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:20.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:20.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:20.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-20 patch sysinv 2020-05-27 15:19:26.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-26 patch sysinv 2020-05-27 15:19:26.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:26.052 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-26 patch sysinv 2020-05-27 15:19:26.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-26 patch sysinv 2020-05-27 15:19:26.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:26.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-26 patch sysinv 2020-05-27 15:19:26.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:26.210 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-26 patch sysinv 2020-05-27 15:19:30.333 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-30 patch sysinv 2020-05-27 15:19:30.334 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:30.469 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:19:30.470 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:19:30.471 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:19:30.471 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:19:30.472 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:19:30.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-30 patch sysinv 2020-05-27 15:19:30.558 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:40.341 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-40 patch sysinv 2020-05-27 15:19:40.342 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:40.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-40 patch sysinv 2020-05-27 15:19:40.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:41.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-41 patch sysinv 2020-05-27 15:19:41.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:41.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-41 patch sysinv 2020-05-27 15:19:41.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-41 patch sysinv 2020-05-27 15:19:41.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:41.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-41 patch sysinv 2020-05-27 15:19:41.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:41.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-41 patch sysinv 2020-05-27 15:19:42.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-42 patch sysinv 2020-05-27 15:19:42.028 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:42.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-42 patch sysinv 2020-05-27 15:19:42.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-42 patch sysinv 2020-05-27 15:19:42.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:42.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-42 patch sysinv 2020-05-27 15:19:42.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:19:42.213 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-19-42 patch sysinv 2020-05-27 15:19:50.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-50 patch sysinv 2020-05-27 15:19:50.361 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:19:50.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-19-50 patch sysinv 2020-05-27 15:19:50.411 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:00.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-00 patch sysinv 2020-05-27 15:20:00.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:00.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-00 patch sysinv 2020-05-27 15:20:00.427 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:01.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-01 patch sysinv 2020-05-27 15:20:01.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:01.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-01 patch sysinv 2020-05-27 15:20:01.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-01 patch sysinv 2020-05-27 15:20:01.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:01.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-01 patch sysinv 2020-05-27 15:20:01.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:01.211 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-01 patch sysinv 2020-05-27 15:20:02.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-02 patch sysinv 2020-05-27 15:20:02.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:02.051 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-02 patch sysinv 2020-05-27 15:20:02.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-02 patch sysinv 2020-05-27 15:20:02.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:02.152 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-02 patch sysinv 2020-05-27 15:20:02.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:02.229 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-02 patch sysinv 2020-05-27 15:20:10.406 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-10 patch sysinv 2020-05-27 15:20:10.407 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:10.450 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-10 patch sysinv 2020-05-27 15:20:10.450 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:17.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-17 patch sysinv 2020-05-27 15:20:17.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:17.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-17 patch sysinv 2020-05-27 15:20:17.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-17 patch sysinv 2020-05-27 15:20:17.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:17.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-17 patch sysinv 2020-05-27 15:20:17.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:17.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-17 patch sysinv 2020-05-27 15:20:20.423 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-20 patch sysinv 2020-05-27 15:20:20.424 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:20.472 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-20 patch sysinv 2020-05-27 15:20:20.473 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:22.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-22 patch sysinv 2020-05-27 15:20:22.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:22.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-22 patch sysinv 2020-05-27 15:20:22.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-22 patch sysinv 2020-05-27 15:20:22.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:22.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-22 patch sysinv 2020-05-27 15:20:22.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:22.707 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-22 patch sysinv 2020-05-27 15:20:30.464 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-30 patch sysinv 2020-05-27 15:20:30.465 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:30.516 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-30 patch sysinv 2020-05-27 15:20:30.517 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:30.541 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:20:30.541 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:20:30.542 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:20:30.543 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:20:30.543 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:20:36.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-36 patch sysinv 2020-05-27 15:20:36.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:36.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-36 patch sysinv 2020-05-27 15:20:36.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-36 patch sysinv 2020-05-27 15:20:36.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:36.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-36 patch sysinv 2020-05-27 15:20:36.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:36.705 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-36 patch sysinv 2020-05-27 15:20:40.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-40 patch sysinv 2020-05-27 15:20:40.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:40.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-40 patch sysinv 2020-05-27 15:20:40.574 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:43.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-43 patch sysinv 2020-05-27 15:20:43.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:43.553 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-43 patch sysinv 2020-05-27 15:20:43.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-43 patch sysinv 2020-05-27 15:20:43.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:43.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-43 patch sysinv 2020-05-27 15:20:43.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:43.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-43 patch sysinv 2020-05-27 15:20:50.486 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-50 patch sysinv 2020-05-27 15:20:50.486 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:50.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-50 patch sysinv 2020-05-27 15:20:50.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:52.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-52 patch sysinv 2020-05-27 15:20:52.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:52.696 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-52 patch sysinv 2020-05-27 15:20:52.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-52 patch sysinv 2020-05-27 15:20:52.736 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:20:52.788 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-20-52 patch sysinv 2020-05-27 15:20:52.789 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:20:52.847 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-20-52 patch sysinv 2020-05-27 15:21:00.508 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-00 patch sysinv 2020-05-27 15:21:00.508 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:00.554 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-00 patch sysinv 2020-05-27 15:21:00.554 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:04.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-04 patch sysinv 2020-05-27 15:21:04.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:04.553 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-04 patch sysinv 2020-05-27 15:21:04.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-04 patch sysinv 2020-05-27 15:21:04.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:04.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-04 patch sysinv 2020-05-27 15:21:04.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:04.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-04 patch sysinv 2020-05-27 15:21:10.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-10 patch sysinv 2020-05-27 15:21:10.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:10.578 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-10 patch sysinv 2020-05-27 15:21:10.579 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:11.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-11 patch sysinv 2020-05-27 15:21:11.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:11.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-11 patch sysinv 2020-05-27 15:21:11.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-11 patch sysinv 2020-05-27 15:21:11.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:11.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-11 patch sysinv 2020-05-27 15:21:11.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:11.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-11 patch sysinv 2020-05-27 15:21:20.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-20 patch sysinv 2020-05-27 15:21:20.550 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:20.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-20 patch sysinv 2020-05-27 15:21:20.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:25.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-25 patch sysinv 2020-05-27 15:21:25.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:25.552 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-25 patch sysinv 2020-05-27 15:21:25.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-25 patch sysinv 2020-05-27 15:21:25.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:25.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-25 patch sysinv 2020-05-27 15:21:25.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:25.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-25 patch sysinv 2020-05-27 15:21:27.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-27 patch sysinv 2020-05-27 15:21:27.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:27.556 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-27 patch sysinv 2020-05-27 15:21:27.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-27 patch sysinv 2020-05-27 15:21:27.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:27.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-27 patch sysinv 2020-05-27 15:21:27.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:27.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-27 patch sysinv 2020-05-27 15:21:30.488 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:21:30.490 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:21:30.491 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:21:30.492 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:21:30.492 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:21:30.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-30 patch sysinv 2020-05-27 15:21:30.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:30.641 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-30 patch sysinv 2020-05-27 15:21:30.641 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:40.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-40 patch sysinv 2020-05-27 15:21:40.591 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:40.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-40 patch sysinv 2020-05-27 15:21:40.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:46.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-46 patch sysinv 2020-05-27 15:21:46.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:46.554 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-46 patch sysinv 2020-05-27 15:21:46.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-46 patch sysinv 2020-05-27 15:21:46.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:46.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-46 patch sysinv 2020-05-27 15:21:46.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:46.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-46 patch sysinv 2020-05-27 15:21:47.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-47 patch sysinv 2020-05-27 15:21:47.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:47.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-47 patch sysinv 2020-05-27 15:21:47.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-47 patch sysinv 2020-05-27 15:21:47.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:47.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-47 patch sysinv 2020-05-27 15:21:47.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:21:47.214 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-21-47 patch sysinv 2020-05-27 15:21:50.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-50 patch sysinv 2020-05-27 15:21:50.619 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:21:50.659 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-21-50 patch sysinv 2020-05-27 15:21:50.659 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:00.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-00 patch sysinv 2020-05-27 15:22:00.632 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:00.678 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-00 patch sysinv 2020-05-27 15:22:00.679 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:02.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-02 patch sysinv 2020-05-27 15:22:02.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:02.051 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-02 patch sysinv 2020-05-27 15:22:02.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-02 patch sysinv 2020-05-27 15:22:02.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:02.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-02 patch sysinv 2020-05-27 15:22:02.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:02.242 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-02 patch sysinv 2020-05-27 15:22:06.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-06 patch sysinv 2020-05-27 15:22:06.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:06.551 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-06 patch sysinv 2020-05-27 15:22:06.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-06 patch sysinv 2020-05-27 15:22:06.599 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:06.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-06 patch sysinv 2020-05-27 15:22:06.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:06.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-06 patch sysinv 2020-05-27 15:22:10.657 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-10 patch sysinv 2020-05-27 15:22:10.658 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:10.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-10 patch sysinv 2020-05-27 15:22:10.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:20.675 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-20 patch sysinv 2020-05-27 15:22:20.676 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:20.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-20 patch sysinv 2020-05-27 15:22:20.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:23.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-23 patch sysinv 2020-05-27 15:22:23.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:23.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-23 patch sysinv 2020-05-27 15:22:23.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-23 patch sysinv 2020-05-27 15:22:23.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:23.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-23 patch sysinv 2020-05-27 15:22:23.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:23.204 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-23 patch sysinv 2020-05-27 15:22:27.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-27 patch sysinv 2020-05-27 15:22:27.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:27.053 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-27 patch sysinv 2020-05-27 15:22:27.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-27 patch sysinv 2020-05-27 15:22:27.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:27.144 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-27 patch sysinv 2020-05-27 15:22:27.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:27.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-27 patch sysinv 2020-05-27 15:22:30.597 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:22:30.598 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:22:30.601 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:22:30.602 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:22:30.602 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:22:30.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-30 patch sysinv 2020-05-27 15:22:30.720 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:30.762 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-30 patch sysinv 2020-05-27 15:22:30.764 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:38.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-38 patch sysinv 2020-05-27 15:22:38.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:38.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-38 patch sysinv 2020-05-27 15:22:38.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-38 patch sysinv 2020-05-27 15:22:38.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:38.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-38 patch sysinv 2020-05-27 15:22:38.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:38.207 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-38 patch sysinv 2020-05-27 15:22:40.714 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-40 patch sysinv 2020-05-27 15:22:40.715 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:40.764 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-40 patch sysinv 2020-05-27 15:22:40.764 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:48.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-48 patch sysinv 2020-05-27 15:22:48.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:48.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-48 patch sysinv 2020-05-27 15:22:48.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-48 patch sysinv 2020-05-27 15:22:48.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:48.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-48 patch sysinv 2020-05-27 15:22:48.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:48.220 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-48 patch sysinv 2020-05-27 15:22:50.886 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-50 patch sysinv 2020-05-27 15:22:50.887 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:50.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-50 patch sysinv 2020-05-27 15:22:50.922 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:57.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-57 patch sysinv 2020-05-27 15:22:57.532 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:57.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-57 patch sysinv 2020-05-27 15:22:57.594 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-57 patch sysinv 2020-05-27 15:22:57.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:22:57.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-22-57 patch sysinv 2020-05-27 15:22:57.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:22:57.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-22-57 patch sysinv 2020-05-27 15:23:00.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-00 patch sysinv 2020-05-27 15:23:00.761 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:00.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-00 patch sysinv 2020-05-27 15:23:00.795 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:08.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-08 patch sysinv 2020-05-27 15:23:08.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:08.547 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-08 patch sysinv 2020-05-27 15:23:08.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-08 patch sysinv 2020-05-27 15:23:08.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:08.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-08 patch sysinv 2020-05-27 15:23:08.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:08.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-08 patch sysinv 2020-05-27 15:23:10.778 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-10 patch sysinv 2020-05-27 15:23:10.779 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:10.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-10 patch sysinv 2020-05-27 15:23:10.828 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:13.539 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-13 patch sysinv 2020-05-27 15:23:13.540 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:13.559 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-13 patch sysinv 2020-05-27 15:23:13.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-13 patch sysinv 2020-05-27 15:23:13.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:13.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-13 patch sysinv 2020-05-27 15:23:13.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:13.707 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-13 patch sysinv 2020-05-27 15:23:20.797 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-20 patch sysinv 2020-05-27 15:23:20.798 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:20.846 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-20 patch sysinv 2020-05-27 15:23:20.846 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:28.565 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-28 patch sysinv 2020-05-27 15:23:28.566 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:28.584 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-28 patch sysinv 2020-05-27 15:23:28.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-28 patch sysinv 2020-05-27 15:23:28.637 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:28.687 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-28 patch sysinv 2020-05-27 15:23:28.688 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:28.746 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-28 patch sysinv 2020-05-27 15:23:30.678 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:23:30.679 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:23:30.682 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:23:30.683 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:23:30.683 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:23:30.850 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-30 patch sysinv 2020-05-27 15:23:30.850 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:30.893 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-30 patch sysinv 2020-05-27 15:23:30.893 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:32.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-32 patch sysinv 2020-05-27 15:23:32.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:32.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-32 patch sysinv 2020-05-27 15:23:32.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-32 patch sysinv 2020-05-27 15:23:32.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:32.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-32 patch sysinv 2020-05-27 15:23:32.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:32.705 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-32 patch sysinv 2020-05-27 15:23:40.845 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-40 patch sysinv 2020-05-27 15:23:40.845 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:40.886 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-40 patch sysinv 2020-05-27 15:23:40.886 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:48.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-48 patch sysinv 2020-05-27 15:23:48.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:48.057 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-48 patch sysinv 2020-05-27 15:23:48.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-48 patch sysinv 2020-05-27 15:23:48.099 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:48.150 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-48 patch sysinv 2020-05-27 15:23:48.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:48.254 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-48 patch sysinv 2020-05-27 15:23:49.527 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-49 patch sysinv 2020-05-27 15:23:49.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:49.547 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-49 patch sysinv 2020-05-27 15:23:49.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-49 patch sysinv 2020-05-27 15:23:49.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:49.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-49 patch sysinv 2020-05-27 15:23:49.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:23:49.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-23-49 patch sysinv 2020-05-27 15:23:50.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-50 patch sysinv 2020-05-27 15:23:50.862 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:23:50.909 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-23-50 patch sysinv 2020-05-27 15:23:50.909 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:00.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-00 patch sysinv 2020-05-27 15:24:00.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:00.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-00 patch sysinv 2020-05-27 15:24:00.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:08.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-08 patch sysinv 2020-05-27 15:24:08.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:08.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-08 patch sysinv 2020-05-27 15:24:08.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-08 patch sysinv 2020-05-27 15:24:08.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:08.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-08 patch sysinv 2020-05-27 15:24:08.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:08.211 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-08 patch sysinv 2020-05-27 15:24:10.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:10.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.599 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:10.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:10.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.899 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.900 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:10.950 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-10 patch sysinv 2020-05-27 15:24:10.951 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:20.933 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-20 patch sysinv 2020-05-27 15:24:20.934 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:21.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-20 patch sysinv 2020-05-27 15:24:21.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:23.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-23 patch sysinv 2020-05-27 15:24:23.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:23.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-23 patch sysinv 2020-05-27 15:24:23.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-23 patch sysinv 2020-05-27 15:24:23.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:23.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-23 patch sysinv 2020-05-27 15:24:23.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:23.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-23 patch sysinv 2020-05-27 15:24:30.760 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:24:30.761 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:24:30.761 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:24:30.762 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:24:30.762 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:24:30.977 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-30 patch sysinv 2020-05-27 15:24:30.978 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:31.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:31.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:31.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:31.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:31.565 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:31.630 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:31.631 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:31.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:31.699 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:31.794 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-31 patch sysinv 2020-05-27 15:24:40.965 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-40 patch sysinv 2020-05-27 15:24:40.965 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:41.011 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-40 patch sysinv 2020-05-27 15:24:41.012 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:43.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-43 patch sysinv 2020-05-27 15:24:43.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:43.052 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-43 patch sysinv 2020-05-27 15:24:43.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-43 patch sysinv 2020-05-27 15:24:43.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:43.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-43 patch sysinv 2020-05-27 15:24:43.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:43.201 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-43 patch sysinv 2020-05-27 15:24:50.989 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-50 patch sysinv 2020-05-27 15:24:50.990 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:51.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-51 patch sysinv 2020-05-27 15:24:51.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:52.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-52 patch sysinv 2020-05-27 15:24:52.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:52.549 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-52 patch sysinv 2020-05-27 15:24:52.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-52 patch sysinv 2020-05-27 15:24:52.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:52.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-52 patch sysinv 2020-05-27 15:24:52.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:52.702 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-52 patch sysinv 2020-05-27 15:24:58.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-58 patch sysinv 2020-05-27 15:24:58.531 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:58.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-58 patch sysinv 2020-05-27 15:24:58.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-58 patch sysinv 2020-05-27 15:24:58.598 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:24:58.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-24-58 patch sysinv 2020-05-27 15:24:58.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:24:58.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-24-58 patch sysinv 2020-05-27 15:25:01.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-00 patch sysinv 2020-05-27 15:25:01.005 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:01.054 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-01 patch sysinv 2020-05-27 15:25:01.054 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:11.026 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-11 patch sysinv 2020-05-27 15:25:11.027 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:11.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-11 patch sysinv 2020-05-27 15:25:11.074 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:13.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-13 patch sysinv 2020-05-27 15:25:13.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:13.548 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-13 patch sysinv 2020-05-27 15:25:13.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-13 patch sysinv 2020-05-27 15:25:13.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:13.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-13 patch sysinv 2020-05-27 15:25:13.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:13.709 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-13 patch sysinv 2020-05-27 15:25:18.533 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-18 patch sysinv 2020-05-27 15:25:18.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:18.554 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-18 patch sysinv 2020-05-27 15:25:18.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-18 patch sysinv 2020-05-27 15:25:18.597 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:18.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-18 patch sysinv 2020-05-27 15:25:18.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:18.703 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-18 patch sysinv 2020-05-27 15:25:21.052 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-21 patch sysinv 2020-05-27 15:25:21.053 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:21.093 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-21 patch sysinv 2020-05-27 15:25:21.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:30.709 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:25:30.710 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:25:30.711 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:25:30.712 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:25:30.712 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:25:31.090 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-31 patch sysinv 2020-05-27 15:25:31.091 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:31.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-31 patch sysinv 2020-05-27 15:25:31.136 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:33.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-33 patch sysinv 2020-05-27 15:25:33.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:33.053 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-33 patch sysinv 2020-05-27 15:25:33.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-33 patch sysinv 2020-05-27 15:25:33.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:33.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-33 patch sysinv 2020-05-27 15:25:33.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:33.206 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-33 patch sysinv 2020-05-27 15:25:34.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-34 patch sysinv 2020-05-27 15:25:34.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:34.051 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-34 patch sysinv 2020-05-27 15:25:34.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-34 patch sysinv 2020-05-27 15:25:34.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:34.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-34 patch sysinv 2020-05-27 15:25:34.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:34.202 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-34 patch sysinv 2020-05-27 15:25:41.091 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-41 patch sysinv 2020-05-27 15:25:41.092 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:41.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-41 patch sysinv 2020-05-27 15:25:41.135 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:51.109 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-51 patch sysinv 2020-05-27 15:25:51.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:51.156 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-51 patch sysinv 2020-05-27 15:25:51.157 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:54.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-54 patch sysinv 2020-05-27 15:25:54.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:54.050 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-54 patch sysinv 2020-05-27 15:25:54.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-54 patch sysinv 2020-05-27 15:25:54.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:54.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-54 patch sysinv 2020-05-27 15:25:54.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:54.207 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-54 patch sysinv 2020-05-27 15:25:55.033 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-55 patch sysinv 2020-05-27 15:25:55.034 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:55.053 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-55 patch sysinv 2020-05-27 15:25:55.096 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-55 patch sysinv 2020-05-27 15:25:55.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:25:55.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-25-55 patch sysinv 2020-05-27 15:25:55.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:25:55.201 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-25-55 patch sysinv 2020-05-27 15:26:01.133 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-01 patch sysinv 2020-05-27 15:26:01.134 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:01.178 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-01 patch sysinv 2020-05-27 15:26:01.178 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:08.535 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-08 patch sysinv 2020-05-27 15:26:08.536 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:08.555 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-08 patch sysinv 2020-05-27 15:26:08.609 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-08 patch sysinv 2020-05-27 15:26:08.610 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:08.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-08 patch sysinv 2020-05-27 15:26:08.662 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:08.882 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-08 patch sysinv 2020-05-27 15:26:11.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-11 patch sysinv 2020-05-27 15:26:11.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:11.197 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-11 patch sysinv 2020-05-27 15:26:11.197 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:16.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-16 patch sysinv 2020-05-27 15:26:16.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:16.048 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-16 patch sysinv 2020-05-27 15:26:16.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-16 patch sysinv 2020-05-27 15:26:16.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:16.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-16 patch sysinv 2020-05-27 15:26:16.145 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:16.200 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-16 patch sysinv 2020-05-27 15:26:21.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-21 patch sysinv 2020-05-27 15:26:21.172 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:21.222 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-21 patch sysinv 2020-05-27 15:26:21.222 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:29.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-29 patch sysinv 2020-05-27 15:26:29.037 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:29.060 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-29 patch sysinv 2020-05-27 15:26:29.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-29 patch sysinv 2020-05-27 15:26:29.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:29.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-29 patch sysinv 2020-05-27 15:26:29.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:29.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-29 patch sysinv 2020-05-27 15:26:30.808 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:26:30.809 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:26:30.810 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:26:30.813 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:26:30.813 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:26:31.210 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-31 patch sysinv 2020-05-27 15:26:31.210 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:31.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-31 patch sysinv 2020-05-27 15:26:31.261 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:36.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-36 patch sysinv 2020-05-27 15:26:36.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:36.557 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-36 patch sysinv 2020-05-27 15:26:36.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-36 patch sysinv 2020-05-27 15:26:36.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:36.645 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-36 patch sysinv 2020-05-27 15:26:36.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:36.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-36 patch sysinv 2020-05-27 15:26:41.213 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-41 patch sysinv 2020-05-27 15:26:41.213 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:41.264 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-41 patch sysinv 2020-05-27 15:26:41.264 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:43.549 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-43 patch sysinv 2020-05-27 15:26:43.550 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:43.588 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-43 patch sysinv 2020-05-27 15:26:43.672 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-43 patch sysinv 2020-05-27 15:26:43.673 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:43.727 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-43 patch sysinv 2020-05-27 15:26:43.728 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:43.797 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-43 patch sysinv 2020-05-27 15:26:51.234 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-51 patch sysinv 2020-05-27 15:26:51.235 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:51.281 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-51 patch sysinv 2020-05-27 15:26:51.281 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:56.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-56 patch sysinv 2020-05-27 15:26:56.534 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:56.554 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-56 patch sysinv 2020-05-27 15:26:56.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-56 patch sysinv 2020-05-27 15:26:56.595 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:26:56.646 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-26-56 patch sysinv 2020-05-27 15:26:56.647 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:26:56.704 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-26-56 patch sysinv 2020-05-27 15:27:01.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-01 patch sysinv 2020-05-27 15:27:01.260 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:01.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-01 patch sysinv 2020-05-27 15:27:01.302 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:04.040 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.041 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:04.070 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.125 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.126 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:04.173 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.174 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:04.236 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.649 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:04.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-27 15:27:04.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock sysinv 2020-05-27 15:27:04.650 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock_controller sysinv 2020-05-27 15:27:04.709 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 host check_lock_worker sysinv 2020-05-27 15:27:04.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action=lock ihost_val_prenotify: {'ihost_action': 'lock'} ihost_val: {'ihost_action': 'lock', 'task': u'Locking'} sysinv 2020-05-27 15:27:04.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'lock'} sysinv 2020-05-27 15:27:04.711 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=lock, notify_vim=True notify_mtce=True rc=True sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action lock sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_lock_action sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: lock sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=True sysinv 2020-05-27 15:27:04.712 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-27 15:27:04.713 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=lock notify_vim=True notify_mtc=True skip_notify_mtce=True sysinv 2020-05-27 15:27:04.713 90934 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'lock'} sysinv 2020-05-27 15:27:04.723 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'lock', 'task': u'Locking'} sysinv 2020-05-27 15:27:04.724 90934 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=lock sysinv 2020-05-27 15:27:04.724 90934 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=lock sysinv 2020-05-27 15:27:04.724 90934 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=lock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'lock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-27 15:27:04.725 90934 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "lock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-27 15:27:04.729 90934 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-27 15:27:04.787 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-04 patch sysinv 2020-05-27 15:27:09.307 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-09 patch sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disable-failed sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disable-failed ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disable-failed, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:27:09.308 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disable-failed sysinv 2020-05-27 15:27:09.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 handle_vim_services_disable_failed ihost_action=lock sysinv 2020-05-27 15:27:09.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-disable-failed sysinv 2020-05-27 15:27:09.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=True sysinv 2020-05-27 15:27:09.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:27:09.309 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=True sysinv 2020-05-27 15:27:09.310 90934 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': '', 'vim_progress_status': '', 'task': ''} sysinv 2020-05-27 15:27:09.319 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': '', 'vim_progress_status': 'services-disable-failed', 'task': ''} sysinv 2020-05-27 15:27:09.377 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-09 patch sysinv 2020-05-27 15:27:11.280 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-11 patch sysinv 2020-05-27 15:27:11.280 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:11.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-11 patch sysinv 2020-05-27 15:27:11.322 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:17.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:17.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.110 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.111 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:17.153 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:17.231 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.528 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.529 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:17.549 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.600 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.600 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:17.829 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:17.830 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:17.903 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-17 patch sysinv 2020-05-27 15:27:21.300 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-21 patch sysinv 2020-05-27 15:27:21.301 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:21.342 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-21 patch sysinv 2020-05-27 15:27:21.343 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:30.895 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:27:30.896 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:27:30.896 90339 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:27:30.897 90339 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 15:27:30.897 90339 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 15:27:31.315 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-31 patch sysinv 2020-05-27 15:27:31.316 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:31.365 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-31 patch sysinv 2020-05-27 15:27:31.366 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:38.031 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-38 patch sysinv 2020-05-27 15:27:38.032 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:38.052 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-38 patch sysinv 2020-05-27 15:27:38.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-38 patch sysinv 2020-05-27 15:27:38.094 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:38.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-38 patch sysinv 2020-05-27 15:27:38.147 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:38.203 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-38 patch sysinv 2020-05-27 15:27:39.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-39 patch sysinv 2020-05-27 15:27:39.030 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:39.049 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-39 patch sysinv 2020-05-27 15:27:39.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-39 patch sysinv 2020-05-27 15:27:39.095 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:39.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-39 patch sysinv 2020-05-27 15:27:39.154 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:39.211 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-39 patch sysinv 2020-05-27 15:27:41.339 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-41 patch sysinv 2020-05-27 15:27:41.340 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:41.385 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-41 patch sysinv 2020-05-27 15:27:41.386 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:51.375 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-51 patch sysinv 2020-05-27 15:27:51.376 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:51.431 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-51 patch sysinv 2020-05-27 15:27:51.431 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:53.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-53 patch sysinv 2020-05-27 15:27:53.530 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:53.550 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-53 patch sysinv 2020-05-27 15:27:53.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-53 patch sysinv 2020-05-27 15:27:53.596 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:53.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-53 patch sysinv 2020-05-27 15:27:53.648 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:53.708 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-53 patch sysinv 2020-05-27 15:27:59.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-59 patch sysinv 2020-05-27 15:27:59.029 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:59.048 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-59 patch sysinv 2020-05-27 15:27:59.097 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-59 patch sysinv 2020-05-27 15:27:59.098 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:27:59.148 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-27-59 patch sysinv 2020-05-27 15:27:59.149 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:27:59.210 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-27-59 patch sysinv 2020-05-27 15:28:01.380 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-01 patch sysinv 2020-05-27 15:28:01.381 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:28:01.426 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-01 patch sysinv 2020-05-27 15:28:01.427 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:28:07.884 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-07 patch sysinv 2020-05-27 15:28:07.885 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-27 15:28:07.885 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_lock_controller sysinv 2020-05-27 15:28:07.909 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action=force-lock ihost_val_prenotify: {'ihost_action': 'force-lock'} ihost_val: {'ihost_action': 'force-lock', 'task': u'Force Locking'} sysinv 2020-05-27 15:28:07.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'force-lock'} sysinv 2020-05-27 15:28:07.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=force-lock, notify_vim=True notify_mtce=False rc=True sysinv 2020-05-27 15:28:07.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=False sysinv 2020-05-27 15:28:07.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action force-lock sysinv 2020-05-27 15:28:07.910 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_force_lock_action sysinv 2020-05-27 15:28:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: force-lock sysinv 2020-05-27 15:28:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=True sysinv 2020-05-27 15:28:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-27 15:28:07.911 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=force-lock notify_vim=True notify_mtc=False skip_notify_mtce=True sysinv 2020-05-27 15:28:07.911 90934 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'force-lock'} sysinv 2020-05-27 15:28:07.920 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'force-lock', 'task': u'Force Locking'} sysinv 2020-05-27 15:28:07.921 90934 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=force-lock sysinv 2020-05-27 15:28:07.921 90934 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=force-lock sysinv 2020-05-27 15:28:07.921 90934 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=force-lock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'force-lock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-27 15:28:07.921 90934 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "force-lock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-27 15:28:07.930 90934 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-27 15:28:07.985 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-07 patch sysinv 2020-05-27 15:28:11.401 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-11 patch sysinv 2020-05-27 15:28:11.402 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:28:11.448 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-11 patch sysinv 2020-05-27 15:28:11.448 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:28:12.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-12 patch sysinv 2020-05-27 15:28:12.981 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:28:12.982 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disable-failed sysinv 2020-05-27 15:28:12.982 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disable-failed ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-27 15:28:12.982 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disable-failed, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-27 15:28:12.982 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:28:12.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disable-failed sysinv 2020-05-27 15:28:12.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 handle_vim_services_disable_failed ihost_action=force-lock sysinv 2020-05-27 15:28:12.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-disable-failed sysinv 2020-05-27 15:28:12.983 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:28:12.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:28:12.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:28:12.984 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-disable-failed'} sysinv 2020-05-27 15:28:12.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 Action services-disable-failed perform notify_mtce sysinv 2020-05-27 15:28:12.994 90934 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-27 15:28:12.995 90934 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "degraded", "invprovision": "provisioned", "peer_id": null, "administrative": "unlocked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "target_load": "19.12", "vsc_controllers": null, "operational": "enabled", "subfunction_avail": "failed", "action": "force-lock", "bm_type": "none"} sysinv 2020-05-27 15:28:13.006 90934 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-27 15:28:13.063 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-13 patch sysinv 2020-05-27 15:28:13.090 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-13 patch sysinv 2020-05-27 15:28:13.091 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:28:13.118 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-13 patch sysinv 2020-05-27 15:28:13.150 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-13 patch sysinv 2020-05-27 15:28:13.151 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-27 15:28:13.189 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-13 patch sysinv 2020-05-27 15:28:13.264 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:28:13.350 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:28:18.810 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-18 patch sysinv 2020-05-27 15:28:18.810 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-27 15:28:18.829 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-18 patch sysinv 2020-05-27 15:28:18.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-18 patch sysinv 2020-05-27 15:28:18.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['administrative'] sysinv 2020-05-27 15:28:18.908 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['administrative'] sysinv 2020-05-27 15:28:18.909 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-27 15:28:18.931 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-18 patch sysinv 2020-05-27 15:28:18.931 90934 INFO sysinv.api.controllers.v1.host [-] Update host memory for (controller-0) sysinv 2020-05-27 15:28:18.931 90934 INFO sysinv.conductor.rpcapi [-] ConductorApi.update_host_memory: sending host memory update request to conductor sysinv 2020-05-27 15:28:18.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-28-18 patch sysinv 2020-05-27 15:28:18.994 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-27 15:28:19.024 90934 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-28-19 patch sysinv 2020-05-27 15:28:19.146 90934 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:28:22.391 90934 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.204.1:5000/v3/auth/tokens: HTTPConnectionPool(host='192.168.204.1', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): ConnectFailure: Unable to establish connection to http://192.168.204.1:5000/v3/auth/tokens: HTTPConnectionPool(host='192.168.204.1', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:37:56.576 10182 INFO sysinv.agent.lldp.manager [-] Configured sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:37:56.579 10182 INFO sysinv.agent.lldp.manager [-] Loaded sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:37:56.579 10182 INFO sysinv.agent.lldp.manager [-] Registered sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:37:56.662 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:37:57.664 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:37:57.671 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:00.673 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:00.679 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:05.681 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:05.687 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:12.689 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:12.695 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:21.699 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:21.705 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:32.711 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:32.717 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:38:45.725 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:38:45.731 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:39:00.736 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:39:00.742 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:39:17.754 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:39:17.760 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:39:36.775 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:39:36.781 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:39:57.789 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:39:57.795 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:06.890 90644 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. sysinv 2020-05-27 15:40:06.892 90644 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". sysinv 2020-05-27 15:40:06.898 90644 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-27 15:40:06.901 90644 INFO oslo.service.wsgi [-] sysinv_api listening on 192.168.204.1:6385 sysinv 2020-05-27 15:40:06.902 90644 INFO oslo_service.service [-] Starting 1 workers fmAPI.cpp(137): Connected to FM Manager. sysinv 2020-05-27 15:40:06.911 90644 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.911 90644 WARNING keystonemiddleware._common.config [-] The option "auth_url" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.911 90644 WARNING keystonemiddleware._common.config [-] The option "project_domain_name" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.911 90644 WARNING keystonemiddleware._common.config [-] The option "username" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.912 90644 WARNING keystonemiddleware._common.config [-] The option "project_name" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.912 90644 WARNING keystonemiddleware._common.config [-] The option "password" in conf is not known to auth_token sysinv 2020-05-27 15:40:06.914 90644 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-27 15:40:06.917 90644 INFO oslo.service.wsgi [-] sysinv_api_pxe listening on 169.254.202.1:6385 sysinv 2020-05-27 15:40:06.917 90644 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-27 15:40:06.990 90481 INFO sysinv.conductor.kube_app [-] Resetting status of app platform-integ-apps from 'applying' to 'uploaded' sysinv 2020-05-27 15:40:07.010 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.011 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.011 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.012 90481 ERROR sysinv.common.kubernetes [-] Failed to delete custom object, Namespace kube-system: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:40:07.012 90481 WARNING sysinv.conductor.kube_app [-] Failed to clear Armada locks.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:40:07.063 90481 INFO sysinv.conductor.manager [-] sysinv-conductor start committed system={'system_mode': u'simplex', 'created_at': datetime.datetime(2020, 4, 21, 9, 57, 41, 963891, tzinfo=), 'uuid': u'1004aa49-f471-4897-97d4-8ba085382f03', 'software_version': u'19.12', 'service_project_name': u'services', 'system_type': u'All-in-one', 'name': u'3609e439-df7e-4046-bcd7-dabd81fb2f18', 'description': None, 'location': None, 'updated_at': datetime.datetime(2020, 4, 21, 9, 59, 7, 193953, tzinfo=), 'capabilities': {u'https_enabled': False, u'vswitch_type': u'none', u'region_config': False, u'sdn_enabled': False, u'shared_services': u'[]'}, 'id': 1, 'contact': None, 'security_feature': u'nopti nospectre_v2', 'services': 72, 'timezone': u'UTC', 'security_profile': u'standard', 'distributed_cloud_role': None, 'region_name': u'RegionOne'} sysinv 2020-05-27 15:40:07.089 90481 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:40:07.095 90481 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-27 15:40:07.126 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.127 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.128 90481 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:40:07.129 90481 ERROR sysinv.conductor.manager [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)). Failed to upgrade/downgrade tiller.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:40:20.814 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:40:20.821 10182 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:40:20.828 10182 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-27 15:40:20.828 10182 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. sysinv 2020-05-27 15:40:21.578 10182 WARNING sysinv.agent.pci [-] Enabling device enp1s0f1 to query link speed sysinv 2020-05-27 15:40:21.585 10182 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp1s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:21.585 10182 WARNING sysinv.agent.pci [-] Disabling device enp1s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:21.794 91084 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. sysinv 2020-05-27 15:40:21.926 10182 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f0 sysinv 2020-05-27 15:40:22.277 10182 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f1 sysinv 2020-05-27 15:40:22.631 10182 WARNING sysinv.agent.pci [-] Enabling device enp134s0f0 to query link speed sysinv 2020-05-27 15:40:22.637 10182 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f0 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:22.638 10182 WARNING sysinv.agent.pci [-] Disabling device enp134s0f0 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:22.980 10182 WARNING sysinv.agent.pci [-] Enabling device enp134s0f1 to query link speed sysinv 2020-05-27 15:40:23.167 10182 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:23.167 10182 WARNING sysinv.agent.pci [-] Disabling device enp134s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:40:23.239 10182 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:40:23.272 90481 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:40:23.299 90481 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'first_report': True} sysinv 2020-05-27 15:40:23.312 10182 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'first_report': True} sysinv 2020-05-27 15:40:23.312 10182 INFO sysinv.agent.manager [-] Agent found matching ihost: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-27 15:40:23.312 10182 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-27 15:40:23.725 90481 INFO sysinv.conductor.manager [-] port 3a2c3d57-f49c-4d28-81e8-0d4172d8024b update attr: {'speed': u'1000', 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.003 90481 INFO sysinv.conductor.manager [-] port 908e869d-e06a-4db2-9861-d571bdc75bc3 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.150 90481 INFO sysinv.conductor.manager [-] port 1fad32fc-c0e3-480d-96de-5281b9d39247 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.311 90481 INFO sysinv.conductor.manager [-] port 8eacf1e6-edc7-4574-ab4b-12f77b656788 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.609 90481 INFO sysinv.conductor.manager [-] port 525c5e74-37ac-4112-be6c-fa2f81da5b40 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.764 90481 INFO sysinv.conductor.manager [-] port 5af31b58-1bdf-4868-944c-1ae0f4f5cc01 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:40:24.838 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ec', 'pclass': u'Unassigned class [ff00]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'ff0000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:24.855 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a1d2', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:24.871 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'-p30', 'pdevice_id': u'a1af', 'pclass': u'USB controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0330', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:24.891 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1b1', 'pclass': u'Signal processing controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'118000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:24.908 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'mei_me', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ba', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:24.925 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1bb', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.071 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1be', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.088 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a182', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.106 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a1', 'pclass': u'Memory controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'058000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.122 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'i801_smbus', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a3', 'pclass': u'SMBus', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0500', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.139 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a4', 'pclass': u'Serial bus controller [0c80]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c8000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.158 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'2000', 'pclass': u'VGA compatible controller', 'psdevice': u'Device 86ed', 'sriov_vfs_pci_address': u'', 'pvendor': u'ASPEED Technology, Inc.', 'pvendor_id': u'1a03', 'pclass_id': u'030000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.175 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'2701', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [Optane] x4 AIC (P4800X)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.190 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.208 90481 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:40:25.276 90481 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0, 1] sysinv 2020-05-27 15:40:25.276 90481 INFO sysinv.conductor.manager [-] Already in db numa_node=1 mynuma_nodes=[0, 1] sysinv 2020-05-27 15:40:25.320 90481 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:2, cores/socket=16, threads/core=2, reference:current (unchanged) sysinv 2020-05-27 15:40:25.320 90481 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 sysinv 2020-05-27 15:40:25.320 90481 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-27 15:40:25.321 90481 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 sysinv 2020-05-27 15:40:25.321 90481 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-27 15:40:25.329 90481 INFO sysinv.conductor.manager [-] update_grub_config, host uuid: (87ae4fec-0a79-4c6d-b963-35cd9d8f0262), force: (True) sysinv 2020-05-27 15:40:25.339 90481 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '6f2216a0-f473-4c79-a2f6-2472673d2ba7'. sysinv 2020-05-27 15:40:25.349 90481 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 6f2216a0-f473-4c79-a2f6-2472673d2ba7, applied 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-27 15:40:25.349 90481 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 4c2d9379-8aa9-4192-b533-9c2979f422bc vs. target: 6f2216a0-f473-4c79-a2f6-2472673d2ba7. sysinv 2020-05-27 15:40:25.351 90481 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:40:25.351 90481 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=6f2216a0-f473-4c79-a2f6-2472673d2ba7, classes: ['platform::compute::grub::runtime', 'platform::compute::config::runtime'] sysinv 2020-05-27 15:40:25.358 90481 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:40:26.386 90481 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-27 15:40:26.386 90481 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-27 15:40:28.107 90481 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-27 15:40:28.109 90481 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-27 15:40:28.110 90481 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 6f2216a0-f473-4c79-a2f6-2472673d2ba7 {'classes': ['platform::compute::grub::runtime', 'platform::compute::config::runtime'], 'force': True, 'personalities': ['controller', 'worker'], 'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} to agent sysinv 2020-05-27 15:40:29.895 90481 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': 0, 'name': u'enp1s0f0'}] sysinv 2020-05-27 15:40:29.900 10182 INFO sysinv.agent.manager [-] Affine platform interface enp1s0f0 with cpulist 0,1 sysinv 2020-05-27 15:40:30.014 10182 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp1s0f0 0,1 return 0 sysinv 2020-05-27 15:40:30.035 10182 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:aceda3e9b5e8 sysinv 2020-05-27 15:40:30.039 90481 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'config_applied': u'4c2d9379-8aa9-4192-b533-9c2979f422bc', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-27 15:40:30.090 90481 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 6f2216a0-f473-4c79-a2f6-2472673d2ba7, applied 4c2d9379-8aa9-4192-b533-9c2979f422bc sysinv 2020-05-27 15:40:30.090 90481 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 4c2d9379-8aa9-4192-b533-9c2979f422bc vs. target: 6f2216a0-f473-4c79-a2f6-2472673d2ba7. sysinv 2020-05-27 15:40:30.096 10182 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': '4c2d9379-8aa9-4192-b533-9c2979f422bc', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-27 15:40:30.470 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-40-30 patch sysinv 2020-05-27 15:40:30.470 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-27 15:40:30.488 91084 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-40-30 patch sysinv 2020-05-27 15:40:30.539 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-40-30 patch sysinv 2020-05-27 15:40:30.539 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail'] sysinv 2020-05-27 15:40:30.558 91084 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-40-30 patch sysinv 2020-05-27 15:40:32.719 10182 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:5e:00.0-nvme-1: 698 ... large disk defaults sysinv 2020-05-27 15:40:32.740 90481 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-05-27 15:40:32.740 90481 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-05-27 15:40:32.740 90481 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-05-27 15:40:32.741 90481 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-05-27 15:40:32.745 10182 INFO sysinv.agent.manager [-] Initial inventory completed host 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-27 15:40:32.764 10182 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 6f2216a0-f473-4c79-a2f6-2472673d2ba7 {u'classes': [u'platform::compute::grub::runtime', u'platform::compute::config::runtime'], u'force': True, u'personalities': [u'controller', u'worker'], u'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} controller sysinv 2020-05-27 15:40:32.764 10182 INFO sysinv.agent.manager [-] controller-active sysinv 2020-05-27 15:40:32.764 10182 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-05-27 15:40:39.476 10182 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'platform::compute::grub::runtime', u'platform::compute::config::runtime']. sysinv 2020-05-27 15:40:39.477 10182 INFO sysinv.agent.manager [-] Agent config applied 6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:40:39.506 90481 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:41:07.254 90481 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '3095c1e1-a400-428a-8e6b-fd67d7fcbe39'. sysinv 2020-05-27 15:41:07.264 90481 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 3095c1e1-a400-428a-8e6b-fd67d7fcbe39, applied 6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:41:07.264 90481 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 6f2216a0-f473-4c79-a2f6-2472673d2ba7 vs. target: 3095c1e1-a400-428a-8e6b-fd67d7fcbe39. sysinv 2020-05-27 15:41:07.274 90481 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:41:07.274 90481 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=3095c1e1-a400-428a-8e6b-fd67d7fcbe39, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'] sysinv 2020-05-27 15:41:07.284 90481 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:41:08.290 90481 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-27 15:41:08.290 90481 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-27 15:41:10.036 90481 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-27 15:41:10.039 90481 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-27 15:41:10.039 90481 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} to agent sysinv 2020-05-27 15:41:10.042 10182 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} controller sysinv 2020-05-27 15:41:10.042 10182 INFO sysinv.agent.manager [-] controller-active sysinv 2020-05-27 15:41:10.042 10182 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-05-27 15:41:10.105 90481 INFO sysinv.conductor.manager [-] drbd-overview: pgsql-20.0, platform-9.8, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribution-16.0 sysinv 2020-05-27 15:41:10.106 90481 INFO sysinv.conductor.manager [-] lvdisplay: pgsql-20.0, platform-10.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribution-16.0 sysinv 2020-05-27 15:41:10.116 90481 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 6f2216a0-f473-4c79-a2f6-2472673d2ba7 sysinv 2020-05-27 15:41:31.645 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-41-31 patch sysinv 2020-05-27 15:41:31.646 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:41:31.646 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disable-failed sysinv 2020-05-27 15:41:31.646 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disable-failed ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-27 15:41:31.646 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disable-failed, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disable-failed sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 handle_vim_services_disable_failed ihost_action=force-lock sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-disable-failed sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:41:31.647 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['vim_progress_status', 'action'] sysinv 2020-05-27 15:41:31.648 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-disable-failed notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:41:31.648 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-disable-failed'} sysinv 2020-05-27 15:41:31.659 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Action services-disable-failed perform notify_mtce sysinv 2020-05-27 15:41:31.659 91084 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-27 15:41:31.659 91084 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "config_status": null, "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "online", "invprovision": "provisioned", "peer_id": null, "administrative": "locked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "task": "", "target_load": "19.12", "vsc_controllers": null, "operational": "disabled", "subfunction_avail": "online", "action": "force-lock", "bm_type": "none"} sysinv 2020-05-27 15:41:31.671 91084 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-27 15:41:31.684 91084 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-41-31 patch sysinv 2020-05-27 15:42:09.988 10182 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime']. sysinv 2020-05-27 15:42:09.989 10182 INFO sysinv.agent.manager [-] Agent config applied 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:42:10.019 90481 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:43:07.428 90481 INFO sysinv.conductor.manager [-] Audit clearing vim_progress_status=services-disable-failed.. sysinv 2020-05-27 15:43:29.918 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-43-29 patch sysinv 2020-05-27 15:43:29.919 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-27 15:43:29.939 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_application sysinv 2020-05-27 15:43:29.942 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_controller sysinv 2020-05-27 15:43:30.236 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost check_unlock_worker sysinv 2020-05-27 15:43:31.045 91084 INFO sysinv.api.controllers.v1.host [-] Memory: Total=96898 MiB, Allocated=7000 MiB, 2M: 44949 pages None pages pending, 1G: 87 pages None pages pending sysinv 2020-05-27 15:43:31.072 91084 INFO sysinv.api.controllers.v1.host [-] Memory: Total=98304 MiB, Allocated=1000 MiB, 2M: 48652 pages None pages pending, 1G: 95 pages None pages pending sysinv 2020-05-27 15:43:31.086 91084 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(1): vm_mem_mib=89898, sysinv 2020-05-27 15:43:31.086 91084 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(1): {'vm_hugepages_nr_4K': 23013888} sysinv 2020-05-27 15:43:31.100 91084 INFO sysinv.api.controllers.v1.host [-] host(controller-0) node(2): vm_mem_mib=97304, sysinv 2020-05-27 15:43:31.101 91084 INFO sysinv.api.controllers.v1.host [-] Updating mem values of host(controller-0) node(2): {'vm_hugepages_nr_4K': 24909824} sysinv 2020-05-27 15:43:31.906 91084 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query_hosts/ hdr:None payload:None sysinv 2020-05-27 15:43:32.265 91084 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'data': [{u'sw_version': u'19.12', u'hostname': u'controller-0', u'nodetype': u'controller', u'patch_failed': False, u'allow_insvc_patching': True, u'subfunctions': [u'controller', u'worker'], u'requires_reboot': False, u'installed': {}, u'state': u'idle', u'interim_state': False, u'secs_since_ack': 15, u'missing_pkgs': [], u'ip': u'192.168.204.2', u'stale_details': False, u'to_remove': [], u'patch_current': True}]} sysinv 2020-05-27 15:43:32.266 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 _semantic_mtc_check_action unlock sysinv 2020-05-27 15:43:32.266 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action=unlock ihost_val_prenotify: {'ihost_action': 'unlock'} ihost_val: {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-27 15:43:32.266 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 host.update.ihost_val_prenotify {'ihost_action': 'unlock'} sysinv 2020-05-27 15:43:32.266 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=unlock, notify_vim=True notify_mtce=True rc=True sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action unlock sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_unlock_action sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: unlock sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:43:32.267 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-27 15:43:32.268 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=unlock notify_vim=True notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:43:32.268 91084 INFO sysinv.api.controllers.v1.host [-] update ihost_val_prenotify: {'ihost_action': 'unlock'} sysinv 2020-05-27 15:43:32.279 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'ihost_action': 'unlock', 'task': u'Unlocking'} sysinv 2020-05-27 15:43:32.279 91084 INFO sysinv.api.controllers.v1.host [-] Notify VIM host action controller-0 action=unlock sysinv 2020-05-27 15:43:32.279 91084 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock sysinv 2020-05-27 15:43:32.279 91084 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_action hostname=controller-0, action=unlock api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'action': 'unlock', 'hostname': u'controller-0', 'uuid': u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262'} sysinv 2020-05-27 15:43:32.280 91084 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"action": "unlock", "hostname": "controller-0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262"} sysinv 2020-05-27 15:43:32.291 91084 INFO sysinv.api.controllers.v1.rest_api [-] Response={} sysinv 2020-05-27 15:43:32.291 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Perform configure_ihost. sysinv 2020-05-27 15:43:32.329 91084 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:43:32.334 90481 INFO sysinv.conductor.manager [-] configure_ihost controller-0 sysinv 2020-05-27 15:43:33.905 90481 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: None sysinv 2020-05-27 15:43:34.801 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-43-34 patch sysinv 2020-05-27 15:43:34.801 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-27 15:43:34.801 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-disabled sysinv 2020-05-27 15:43:34.802 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-disabled ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-27 15:43:34.802 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-disabled, notify_vim=False notify_mtce=True rc=True sysinv 2020-05-27 15:43:34.802 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-disabled notify_vim=False notify_mtc=True skip_notify_mtce=False sysinv 2020-05-27 15:43:34.802 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-disabled sysinv 2020-05-27 15:43:34.802 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 _handle_vim_services_disabled' sysinv 2020-05-27 15:43:34.802 91084 WARNING sysinv.api.controllers.v1.host [-] controller-0 Admin action task not Locking or Force Locking upon receipt of 'services-disabled'. sysinv 2020-05-27 15:43:34.803 91084 WARNING sysinv.api.controllers.v1.host [-] controller-0 Exit _handle_vim_services_disabled sysinv 2020-05-27 15:43:34.803 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action aborting request services-disabled ['action'] sysinv 2020-05-27 15:43:34.803 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end stage_action rc exit_return_host sysinv 2020-05-27 15:43:34.960 90481 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-27 15:43:34.960 90481 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-27 15:43:35.885 90481 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-27 15:43:35.922 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 Action unlock perform notify_mtce sysinv 2020-05-27 15:43:35.923 91084 INFO sysinv.api.controllers.v1.mtce_api [-] number of calls to rest_api_request=1 (max_retry=3) sysinv 2020-05-27 15:43:35.923 91084 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:2112/v1/hosts/87ae4fec-0a79-4c6d-b963-35cd9d8f0262 hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot": "false", "ttys_dcd": null, "subfunctions": "controller,worker", "bm_ip": null, "install_state": null, "rootfs_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 1, "vim_progress_status": "", "console": "tty0", "uuid": "87ae4fec-0a79-4c6d-b963-35cd9d8f0262", "mgmt_ip": "192.168.204.2", "software_load": "19.12", "config_status": null, "hostname": "controller-0", "iscsi_initiator_name": "iqn.1994-05.com.redhat:aceda3e9b5e8", "capabilities": {"stor_function": "monitor"}, "install_output": "text", "location": {}, "availability": "online", "invprovision": "provisioned", "peer_id": null, "administrative": "locked", "personality": "controller", "recordtype": "standard", "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "1004aa49-f471-4897-97d4-8ba085382f03", "boot_device": "/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "install_state_info": null, "mgmt_mac": "00:00:00:00:00:00", "subfunction_oper": "disabled", "target_load": "19.12", "vsc_controllers": null, "operational": "disabled", "subfunction_avail": "online", "action": "unlock", "bm_type": "none"} sysinv 2020-05-27 15:43:35.934 91084 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'pass'} sysinv 2020-05-27 15:43:35.950 91084 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-43-35 patch sysinv 2020-05-27 15:43:35.981 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-43-35 patch sysinv 2020-05-27 15:43:35.982 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_avail', 'administrative', 'availability'] sysinv 2020-05-27 15:43:35.982 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['administrative'] sysinv 2020-05-27 15:43:35.982 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=None notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-27 15:43:36.003 91084 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-43-36 patch sysinv 2020-05-27 15:43:36.027 91084 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:43:49.859 90644 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-27 15:43:49.860 90644 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-27 15:43:49.860 90644 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-27 15:43:49.860 91084 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-27 15:43:49.873 90644 INFO oslo_service.service [-] Child 91084 exited with status 0 sysinv 2020-05-27 15:43:49.874 90644 INFO oslo_service.service [-] Caught SIGTERM, stopping children sysinv 2020-05-27 15:43:49.874 90644 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-27 15:43:49.874 90644 INFO oslo_service.service [-] Waiting on 1 children to exit sysinv 2020-05-27 15:43:49.875 91085 INFO oslo.service.wsgi [-] Stopping WSGI server. sysinv 2020-05-27 15:43:49.879 90644 INFO oslo_service.service [-] Child 91085 exited with status 0 sysinv 2020-05-27 15:43:51.353 90481 INFO sysinv.openstack.common.service [-] Caught SIGTERM, exiting sysinv 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 104] Connection reset by peer: error: [Errno 104] Connection reset by peer 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common Traceback (most recent call last): 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 564, in ensure 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return method(*args, **kwargs) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/impl_kombu.py", line 644, in _consume 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return connection.drain_events(**kwargs) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 464, in drain_events 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return self.blocking_read(timeout) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 469, in blocking_read 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return self.on_inbound_frame(frame) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 68, in on_frame 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common callback(channel, method_sig, buf, None) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 473, in on_inbound_method 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common method_sig, payload, content, 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common listener(*args) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 593, in _on_close 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common self._x_close_ok() 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 608, in _x_close_ok 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 64, in send_method 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common conn.frame_writer(1, self.channel_id, sig, args, content) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 174, in write_frame 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common write(view[:offset]) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 301, in write 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common self._write(s) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common tail = self.send(data, flags) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return self._send_loop(self.fd.send, data, flags) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common return send_method(data, *args) 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common error: [Errno 104] Connection reset by peer 2020-05-27 15:43:54.851 10182 ERROR sysinv.openstack.common.rpc.common sysinv 2020-05-27 15:43:54.854 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:43:54.860 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:43:55.862 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:43:55.868 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:43:58.872 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:43:58.878 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:03.881 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:03.887 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:10.049 10182 ERROR sysinv.openstack.common.rpc.common [-] Failed to declare consumer for topic 'dafc9055ee244358bd22f5d9cb562d81': [Errno 104] Connection reset by peer: error: [Errno 104] Connection reset by peer sysinv 2020-05-27 15:44:10.049 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:10.056 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:10.888 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:10.894 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:11.056 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:11.062 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:14.063 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:14.069 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:19.072 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:19.079 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:44:19.896 10182 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:44:19.901 10182 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:49:35.105 10095 INFO sysinv.agent.lldp.manager [-] Configured sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:49:35.112 10095 INFO sysinv.agent.lldp.manager [-] Loaded sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:49:35.112 10095 INFO sysinv.agent.lldp.manager [-] Registered sysinv LLDP agent drivers: ['lldpd'] sysinv 2020-05-27 15:49:35.169 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:49:36.170 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:49:36.176 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:49:39.179 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:49:39.185 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:49:44.187 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:49:44.193 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:49:51.195 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:49:51.201 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:50:00.205 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:50:00.211 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:50:11.217 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:50:11.223 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:50:24.231 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:50:24.237 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:50:39.248 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:50:39.254 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:50:56.266 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:50:56.272 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:15.288 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:51:15.294 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:36.310 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:51:36.316 10095 ERROR sysinv.openstack.common.rpc.common [-] AMQP server on 192.168.204.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.: error: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.317 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.318 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.318 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.319 90332 ERROR sysinv.common.kubernetes [-] Failed to delete custom object, Namespace kube-system: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:51:46.319 90332 WARNING sysinv.conductor.kube_app [-] Failed to clear Armada locks.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /apis/armada.process/v1/namespaces/kube-system/locks/locks.armada.process.lock (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:51:46.326 90487 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future. sysinv 2020-05-27 15:51:46.328 90487 WARNING oslo_config.cfg [-] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken". sysinv 2020-05-27 15:51:46.334 90487 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-27 15:51:46.337 90487 INFO oslo.service.wsgi [-] sysinv_api listening on 192.168.204.1:6385 sysinv 2020-05-27 15:51:46.338 90487 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-27 15:51:46.346 90487 WARNING keystonemiddleware._common.config [-] The option "user_domain_name" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.347 90487 WARNING keystonemiddleware._common.config [-] The option "auth_url" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.347 90487 WARNING keystonemiddleware._common.config [-] The option "project_domain_name" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.347 90487 WARNING keystonemiddleware._common.config [-] The option "username" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.348 90487 WARNING keystonemiddleware._common.config [-] The option "project_name" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.348 90487 WARNING keystonemiddleware._common.config [-] The option "password" in conf is not known to auth_token sysinv 2020-05-27 15:51:46.350 90487 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. sysinv 2020-05-27 15:51:46.353 90487 INFO oslo.service.wsgi [-] sysinv_api_pxe listening on 169.254.202.1:6385 sysinv 2020-05-27 15:51:46.353 90487 INFO oslo_service.service [-] Starting 1 workers sysinv 2020-05-27 15:51:46.371 90332 INFO sysinv.conductor.manager [-] sysinv-conductor start committed system={'system_mode': u'simplex', 'created_at': datetime.datetime(2020, 4, 21, 9, 57, 41, 963891, tzinfo=), 'uuid': u'1004aa49-f471-4897-97d4-8ba085382f03', 'software_version': u'19.12', 'service_project_name': u'services', 'system_type': u'All-in-one', 'name': u'3609e439-df7e-4046-bcd7-dabd81fb2f18', 'description': None, 'location': None, 'updated_at': datetime.datetime(2020, 4, 21, 9, 59, 7, 193953, tzinfo=), 'capabilities': {u'https_enabled': False, u'vswitch_type': u'none', u'region_config': False, u'sdn_enabled': False, u'shared_services': u'[]'}, 'id': 1, 'contact': None, 'security_feature': u'nopti nospectre_v2', 'services': 72, 'timezone': u'UTC', 'security_profile': u'standard', 'distributed_cloud_role': None, 'region_name': u'RegionOne'} sysinv 2020-05-27 15:51:46.397 90332 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:51:46.403 90332 INFO sysinv.conductor.manager [-] _upgrade_downgrade_tiller sysinv 2020-05-27 15:51:46.431 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.435 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.436 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 15:51:46.437 90332 ERROR sysinv.conductor.manager [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)). Failed to upgrade/downgrade tiller.: MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/pods?labelSelector=name%3Dtiller (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 15:51:59.334 10095 INFO sysinv.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:51:59.346 10095 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:51:59.354 10095 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-27 15:51:59.354 10095 INFO sysinv.agent.manager [-] Sysinv Agent audit running inv_get_and_report. sysinv 2020-05-27 15:52:00.095 10095 WARNING sysinv.agent.pci [-] Enabling device enp1s0f1 to query link speed sysinv 2020-05-27 15:52:00.103 10095 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp1s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:00.103 10095 WARNING sysinv.agent.pci [-] Disabling device enp1s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:00.434 10095 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f0 sysinv 2020-05-27 15:52:00.765 10095 ERROR sysinv.agent.pci [-] Invalid port speed = -1 for enp24s0f1 sysinv 2020-05-27 15:52:01.094 10095 WARNING sysinv.agent.pci [-] Enabling device enp134s0f0 to query link speed sysinv 2020-05-27 15:52:01.100 10095 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f0 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:01.100 10095 WARNING sysinv.agent.pci [-] Disabling device enp134s0f0 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:01.182 90935 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. sysinv 2020-05-27 15:52:01.430 10095 WARNING sysinv.agent.pci [-] Enabling device enp134s0f1 to query link speed sysinv 2020-05-27 15:52:01.628 10095 WARNING sysinv.agent.pci [-] ATTR speed unknown for: enp134s0f1 (flags: 0x1002): IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:01.628 10095 WARNING sysinv.agent.pci [-] Disabling device enp134s0f1 after querying link speed: IOError: [Errno 22] Invalid argument sysinv 2020-05-27 15:52:01.695 10095 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:52:01.717 90332 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:52:01.740 90332 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'first_report': True} sysinv 2020-05-27 15:52:01.752 10095 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'first_report': True} sysinv 2020-05-27 15:52:01.752 10095 INFO sysinv.agent.manager [-] Agent found matching ihost: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-27 15:52:01.753 10095 INFO sysinv.agent.manager [-] _report_to_conductor initial_reports_required=set(['pci_device', 'pv', 'cpu', 'lvg', 'numa', 'memory', 'disk', 'port']) sysinv 2020-05-27 15:52:02.200 90332 INFO sysinv.conductor.manager [-] port 3a2c3d57-f49c-4d28-81e8-0d4172d8024b update attr: {'speed': u'1000', 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:02.473 90332 INFO sysinv.conductor.manager [-] port 908e869d-e06a-4db2-9861-d571bdc75bc3 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'7\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'igb', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:02.619 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:02.619 90332 INFO sysinv.conductor.manager [-] port 1fad32fc-c0e3-480d-96de-5281b9d39247 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:02.745 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-52-02 patch sysinv 2020-05-27 15:52:02.745 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:52:02.764 90332 INFO sysinv.conductor.manager [-] port 8eacf1e6-edc7-4574-ab4b-12f77b656788 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:03.027 90332 INFO sysinv.conductor.manager [-] port 525c5e74-37ac-4112-be6c-fa2f81da5b40 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:03.165 90332 INFO sysinv.conductor.manager [-] port 5af31b58-1bdf-4868-944c-1ae0f4f5cc01 update attr: {'speed': None, 'sriov_vfs_pci_address': u'', 'sriov_totalvfs': u'63\n', 'sriov_vf_pdevice_id': None, 'dpdksupport': True, 'sriov_numvfs': u'0\n', 'driver': u'ixgbe', 'sriov_vf_driver': None} sysinv 2020-05-27 15:52:03.226 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ec', 'pclass': u'Unassigned class [ff00]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'ff0000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.242 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a1d2', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.258 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'-p30', 'pdevice_id': u'a1af', 'pclass': u'USB controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0330', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.275 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1b1', 'pclass': u'Signal processing controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'118000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.290 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'mei_me', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1ba', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.305 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1bb', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.431 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1be', 'pclass': u'Communication controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'078000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.447 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'ahci', 'psvendor': u'-p01', 'pdevice_id': u'a182', 'pclass': u'SATA controller', 'psdevice': u'ASUSTeK Computer Inc.', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010601', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.463 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a1', 'pclass': u'Memory controller', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'058000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.478 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'i801_smbus', 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a3', 'pclass': u'SMBus', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c0500', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.494 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'a1a4', 'pclass': u'Serial bus controller [0c80]', 'psdevice': u'Device 871e', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'0c8000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.510 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': None, 'psvendor': u'ASUSTeK Computer Inc.', 'pdevice_id': u'2000', 'pclass': u'VGA compatible controller', 'psdevice': u'Device 86ed', 'sriov_vfs_pci_address': u'', 'pvendor': u'ASPEED Technology, Inc.', 'pvendor_id': u'1a03', 'pclass_id': u'030000', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.525 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'2701', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [Optane] x4 AIC (P4800X)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.540 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.558 90332 INFO sysinv.conductor.manager [-] attr: {'sriov_numvfs': 0, 'driver': u'nvme', 'psvendor': u'Intel Corporation', 'pdevice_id': u'0a54', 'pclass': u'Non-Volatile memory controller', 'psdevice': u'NVMe Datacenter SSD [3DNAND] ME 2.5" U.2 (P4610)', 'sriov_vfs_pci_address': u'', 'pvendor': u'Intel Corporation', 'pvendor_id': u'8086', 'pclass_id': u'010802', 'sriov_totalvfs': None} sysinv 2020-05-27 15:52:03.623 90332 INFO sysinv.conductor.manager [-] Already in db numa_node=0 mynuma_nodes=[0, 1] sysinv 2020-05-27 15:52:03.623 90332 INFO sysinv.conductor.manager [-] Already in db numa_node=1 mynuma_nodes=[0, 1] sysinv 2020-05-27 15:52:03.665 90332 INFO sysinv.conductor.manager [-] Logical CPU topology: host:controller-0 (controller,worker), sockets:2, cores/socket=16, threads/core=2, reference:current (unchanged) sysinv 2020-05-27 15:52:03.665 90332 INFO sysinv.conductor.manager [-] cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 sysinv 2020-05-27 15:52:03.665 90332 INFO sysinv.conductor.manager [-] socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-27 15:52:03.665 90332 INFO sysinv.conductor.manager [-] core_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 sysinv 2020-05-27 15:52:03.665 90332 INFO sysinv.conductor.manager [-] thread_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 sysinv 2020-05-27 15:52:04.178 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-52-04 patch sysinv 2020-05-27 15:52:04.178 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-27 15:52:04.197 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-52-04 patch sysinv 2020-05-27 15:52:04.250 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:05.363 90332 INFO sysinv.conductor.manager [-] platform_interfaces host_id=1 info_list=[{'numa_node': 0, 'name': u'enp1s0f0'}] sysinv 2020-05-27 15:52:05.368 10095 INFO sysinv.agent.manager [-] Affine platform interface enp1s0f0 with cpulist 0,1 sysinv 2020-05-27 15:52:05.474 10095 INFO sysinv.agent.manager [-] /usr/bin/affine-interrupts.sh enp1s0f0 0,1 return 0 sysinv 2020-05-27 15:52:05.496 10095 INFO sysinv.agent.manager [-] iscsi initiator name = iqn.1994-05.com.redhat:aceda3e9b5e8 sysinv 2020-05-27 15:52:05.500 90332 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'config_applied': u'3095c1e1-a400-428a-8e6b-fd67d7fcbe39', u'first_report': True, u'availability': u'available', u'iscsi_initiator_name': u'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-27 15:52:05.566 90332 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 fmAPI.cpp(137): Connected to FM Manager. sysinv 2020-05-27 15:52:05.582 10095 INFO sysinv.agent.manager [-] Sysinv Agent platform update by host: {'config_applied': '3095c1e1-a400-428a-8e6b-fd67d7fcbe39', 'first_report': True, 'availability': 'available', 'iscsi_initiator_name': 'iqn.1994-05.com.redhat:aceda3e9b5e8'} sysinv 2020-05-27 15:52:08.340 10095 INFO sysinv.common.utils [-] Disk size for /dev/disk/by-path/pci-0000:5e:00.0-nvme-1: 698 ... large disk defaults sysinv 2020-05-27 15:52:08.364 90332 INFO sysinv.conductor.manager [-] Host FS 'scratch' already exists sysinv 2020-05-27 15:52:08.365 90332 INFO sysinv.conductor.manager [-] Host FS 'backup' already exists sysinv 2020-05-27 15:52:08.365 90332 INFO sysinv.conductor.manager [-] Host FS 'docker' already exists sysinv 2020-05-27 15:52:08.365 90332 INFO sysinv.conductor.manager [-] Host FS 'kubelet' already exists sysinv 2020-05-27 15:52:08.369 10095 INFO sysinv.agent.manager [-] Initial inventory completed host 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 sysinv 2020-05-27 15:52:19.353 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:19.419 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:19.841 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:19.908 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-52-19 patch sysinv 2020-05-27 15:52:19.908 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 15:52:19.983 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-52-19 patch sysinv 2020-05-27 15:52:20.031 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:52:47.363 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:52:47.374 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940962027664'} sysinv 2020-05-27 15:52:47.380 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:52:47.387 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":501,"data_bytes":1659566835,"bytes_used":1761845248,"bytes_avail":1596703776768,"bytes_total":1598465622016},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928057808'} sysinv 2020-05-27 15:52:47.404 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-05-27 15:52:47.667 90332 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '3afd9bfa-7367-47f9-a103-9749a99744fc'. sysinv 2020-05-27 15:52:47.678 90332 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 3afd9bfa-7367-47f9-a103-9749a99744fc, applied 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:52:47.678 90332 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 vs. target: 3afd9bfa-7367-47f9-a103-9749a99744fc. sysinv 2020-05-27 15:52:47.732 90332 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=3afd9bfa-7367-47f9-a103-9749a99744fc sysinv 2020-05-27 15:52:47.732 90332 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=3afd9bfa-7367-47f9-a103-9749a99744fc, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'] sysinv 2020-05-27 15:52:47.739 90332 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 3afd9bfa-7367-47f9-a103-9749a99744fc sysinv 2020-05-27 15:52:48.769 90332 INFO sysinv.puppet.interface [-] Interface data0 has no primary address sysinv 2020-05-27 15:52:48.769 90332 INFO sysinv.puppet.interface [-] Interface data1 has no primary address sysinv 2020-05-27 15:52:50.210 90332 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-27 15:52:50.212 90332 INFO sysinv.conductor.manager [-] stx-openstack app status does not warrant re-apply sysinv 2020-05-27 15:52:50.213 90332 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 3afd9bfa-7367-47f9-a103-9749a99744fc {'classes': ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller'], 'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} to agent sysinv 2020-05-27 15:52:50.216 10095 INFO sysinv.agent.manager [-] config_apply_runtime_manifest: 3afd9bfa-7367-47f9-a103-9749a99744fc {u'classes': [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime'], u'force': False, u'personalities': [u'controller'], u'host_uuids': [u'87ae4fec-0a79-4c6d-b963-35cd9d8f0262']} controller sysinv 2020-05-27 15:52:50.216 10095 INFO sysinv.agent.manager [-] controller-active sysinv 2020-05-27 15:52:50.216 10095 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/19.12/hieradata' sysinv 2020-05-27 15:52:50.276 90332 INFO sysinv.conductor.manager [-] drbd-overview: pgsql-20.0, platform-9.8, extension-0.96875, patch-vault-0, etcd-4.8, dockerdistribution-16.0 sysinv 2020-05-27 15:52:50.276 90332 INFO sysinv.conductor.manager [-] lvdisplay: pgsql-20.0, platform-10.0, extension-1.0, patch-vault-0, etcd-5.0, dockerdistribution-16.0 sysinv 2020-05-27 15:52:50.285 90332 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 3095c1e1-a400-428a-8e6b-fd67d7fcbe39 sysinv 2020-05-27 15:53:06.880 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['uptime', 'task'] sysinv 2020-05-27 15:53:07.000 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-53-06 patch sysinv 2020-05-27 15:53:07.001 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 15:53:07.053 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-53-07 patch sysinv 2020-05-27 15:53:07.054 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-27 15:53:07.224 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-53-07 patch sysinv 2020-05-27 15:53:34.295 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-53-34 patch sysinv 2020-05-27 15:53:34.295 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-27 15:53:34.355 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-53-34 patch sysinv 2020-05-27 15:53:46.606 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:53:46.614 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940928187152'} sysinv 2020-05-27 15:53:46.619 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:53:46.626 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":501,"data_bytes":1659566835,"bytes_used":1761943552,"bytes_avail":1596703678464,"bytes_total":1598465622016,"read_bytes_sec":209079,"write_bytes_sec":74013,"read_op_per_sec":18,"write_op_per_sec":12},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928221584'} sysinv 2020-05-27 15:53:46.641 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-05-27 15:53:48.709 10095 INFO sysinv.agent.manager [-] Runtime manifest apply completed for classes [u'openstack::keystone::endpoint::runtime', u'platform::firewall::runtime']. sysinv 2020-05-27 15:53:48.710 10095 INFO sysinv.agent.manager [-] Agent config applied 3afd9bfa-7367-47f9-a103-9749a99744fc sysinv 2020-05-27 15:53:48.740 90332 INFO sysinv.conductor.manager [-] SYS_I Clear system config alarm: controller-0 target config 3afd9bfa-7367-47f9-a103-9749a99744fc sysinv 2020-05-27 15:54:46.757 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:54:46.765 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940928122192'} sysinv 2020-05-27 15:54:46.770 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:54:46.777 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":501,"data_bytes":1659808499,"bytes_used":1762353152,"bytes_avail":1596703268864,"bytes_total":1598465622016,"read_bytes_sec":70284,"write_bytes_sec":47424,"read_op_per_sec":4,"write_op_per_sec":5},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928186384'} sysinv 2020-05-27 15:54:46.790 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-05-27 15:55:46.656 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:55:46.664 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940928184656'} sysinv 2020-05-27 15:55:46.669 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:55:46.675 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":502,"data_bytes":1659828979,"bytes_used":1762799616,"bytes_avail":1596702822400,"bytes_total":1598465622016,"read_bytes_sec":89384,"write_bytes_sec":108489,"read_op_per_sec":6,"write_op_per_sec":20},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928122832'} sysinv 2020-05-27 15:55:46.689 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. sysinv 2020-05-27 15:56:09.956 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-56-09 patch sysinv 2020-05-27 15:56:09.957 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['action'] sysinv 2020-05-27 15:56:09.957 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 Pending update_vim_progress_status services-enabled sysinv 2020-05-27 15:56:09.957 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 action=services-enabled ihost_val_prenotify: {} ihost_val: {} sysinv 2020-05-27 15:56:09.957 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 action_check action=services-enabled, notify_vim=False notify_mtce=False rc=True sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_check hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=False sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 stage_action services-enabled sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 received services-enabled task= vim_progress_status= sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 Action staged: services-enabled sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 post action_stage hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-27 15:56:09.958 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 2. delta_handle ['action'] sysinv 2020-05-27 15:56:09.959 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 post delta_handle hostupdate action=services-enabled notify_vim=False notify_mtc=False skip_notify_mtce=True sysinv 2020-05-27 15:56:09.959 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 apply ihost_val {'vim_progress_status': 'services-enabled'} sysinv 2020-05-27 15:56:10.022 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 notify_availability=services-enabled sysinv 2020-05-27 15:56:10.059 90935 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-05-27 15:56:10.064 90332 INFO sysinv.conductor.manager [-] Updating platform data for host: 87ae4fec-0a79-4c6d-b963-35cd9d8f0262 with: {u'availability': u'services-enabled'} sysinv 2020-05-27 15:56:10.114 90332 INFO sysinv.conductor.manager [-] platform-integ-apps app status does not warrant re-apply sysinv 2020-05-27 15:56:10.119 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-56-10 patch sysinv 2020-05-27 15:56:14.740 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-15-56-14 patch sysinv 2020-05-27 15:56:14.740 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability'] sysinv 2020-05-27 15:56:14.808 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-15-56-14 patch sysinv 2020-05-27 15:56:46.699 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 15:56:46.708 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940928224912'} sysinv 2020-05-27 15:56:46.713 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 15:56:46.720 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":505,"data_bytes":1659890419,"bytes_used":1763422208,"bytes_avail":1596702199808,"bytes_total":1598465622016,"read_bytes_sec":8528,"write_bytes_sec":163405,"read_op_per_sec":0,"write_op_per_sec":33},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928119632'} sysinv 2020-05-27 15:56:47.045 90332 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 15:56:47.405 90332 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 15:56:47.412 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 15:56:47.586 90332 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 15:56:47.887 90332 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 15:56:48.011 90332 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 15:56:48.177 90332 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 15:56:48.178 90332 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 15:56:48.238 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:56:48.240 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:56:48.278 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:56:48.279 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:56:48.279 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:56:48.300 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:57:18.296 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:57:18.298 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:57:18.331 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:57:18.332 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:57:18.332 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 15:57:18.360 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:57:48.333 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 15:57:48.335 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 15:57:48.371 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:57:48.372 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 15:57:48.372 90332 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 15:57:48.518 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 15:57:48.521 90332 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 15:57:48.522 90332 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 16:03:07.109 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-03-07 patch sysinv 2020-05-27 16:03:07.110 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:03:47.075 90332 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 16:03:47.097 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 16:03:47.106 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940962027920'} sysinv 2020-05-27 16:03:47.112 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 16:03:47.122 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":515,"data_bytes":1660599027,"bytes_used":1764106240,"bytes_avail":1596701515776,"bytes_total":1598465622016,"write_bytes_sec":94800,"read_op_per_sec":0,"write_op_per_sec":19},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928233680'} sysinv 2020-05-27 16:03:47.511 90332 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 16:03:47.592 90332 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 16:03:47.603 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 16:03:47.809 90332 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 16:03:48.109 90332 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 16:03:48.152 90332 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 16:03:48.467 90332 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 16:03:48.468 90332 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 16:03:48.551 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:03:48.553 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:03:48.594 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:03:48.595 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:03:48.595 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:03:48.620 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:04:15.398 90935 WARNING wsme.api [-] Client-side error: Application-apply rejected: install/update is already in progress.: ClientSideError: Application-apply rejected: install/update is already in progress. sysinv 2020-05-27 16:04:18.616 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:04:18.619 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:04:18.658 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:04:18.659 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:04:18.659 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:04:18.686 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:04:28.558 90332 INFO sysinv.conductor.kube_app [-] Abort set for app platform-integ-apps sysinv 2020-05-27 16:04:48.660 90332 INFO sysinv.conductor.kube_app [-] User aborted. Skipping download of image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: operation aborted by user.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: operation aborted by user. 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 728, in _download_images 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app reason="operation aborted by user.") 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: operation aborted by user. 2020-05-27 16:04:48.661 90332 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 16:04:48.802 90332 INFO sysinv.conductor.kube_app [-] User aborted. Skipping download of image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 sysinv 2020-05-27 16:04:48.842 90332 INFO sysinv.conductor.kube_app [-] Application apply aborted by user!. sysinv 2020-05-27 16:04:48.842 90332 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 16:05:03.400 90332 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 16:05:03.746 90332 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 16:05:03.777 90332 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 16:05:03.971 90332 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 16:05:03.971 90332 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 16:05:04.045 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:05:04.048 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:05:04.093 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:05:04.094 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:05:04.094 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:05:04.116 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:05:34.124 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:05:34.126 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:05:34.164 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:05:34.165 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:05:34.165 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:05:34.187 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:06:04.180 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:06:04.183 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:06:04.218 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:06:04.219 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:06:04.220 90332 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 16:06:04.370 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:06:04.375 90332 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 16:06:04.376 90332 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last): 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp **args) 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10467, in perform_app_apply 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode) 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp self._download_images(app) 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:06:04.376 90332 ERROR sysinv.openstack.common.rpc.amqp sysinv 2020-05-27 16:08:07.192 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-08-07 patch sysinv 2020-05-27 16:08:07.192 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:11:47.380 90332 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 16:11:47.408 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 16:11:47.419 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940962027600'} sysinv 2020-05-27 16:11:47.427 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 16:11:47.438 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":515,"data_bytes":1661442803,"bytes_used":1765175296,"bytes_avail":1596700446720,"bytes_total":1598465622016,"write_bytes_sec":91403,"read_op_per_sec":0,"write_op_per_sec":19},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928119824'} sysinv 2020-05-27 16:11:47.875 90332 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 16:11:47.954 90332 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 16:11:47.964 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 16:11:48.154 90332 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 16:11:48.453 90332 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 16:11:48.505 90332 INFO sysinv.conductor.kube_app [-] Generating application overrides... sysinv 2020-05-27 16:11:48.778 90332 INFO sysinv.helm.manifest_base [-] Delete manifest file /opt/platform/armada/19.12/platform-integ-apps/1.0-8/platform-integ-apps-manifest-del.yaml generated sysinv 2020-05-27 16:11:48.778 90332 INFO sysinv.conductor.kube_app [-] Application overrides generated. sysinv 2020-05-27 16:11:48.852 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:11:48.853 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:11:48.892 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:11:48.893 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:11:48.893 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:11:48.957 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:12:18.914 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:12:18.917 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:12:18.962 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:12:18.963 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:12:18.963 90332 INFO sysinv.conductor.kube_app [-] Retry docker images download for application platform-integ-apps after 30 seconds sysinv 2020-05-27 16:12:18.982 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:12:48.965 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download started from local registry sysinv 2020-05-27 16:12:48.967 90332 INFO sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download started from local registry sysinv 2020-05-27 16:12:49.013 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/quay.io/external_storage/rbd-provisioner/manifests/v2.1.1-k8s1.11: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Aquay.io%2Fexternal_storage%2Frbd-provisioner%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:12:49.016 90332 INFO sysinv.conductor.kube_app [-] Failed to download image: registry.local:9001/quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11 sysinv 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app [-] Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s).: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1979, in perform_app_apply 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app self._download_images(app) 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 748, in _download_images 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED) 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). 2020-05-27 16:12:49.016 90332 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 16:12:49.171 90332 ERROR sysinv.conductor.kube_app [-] Image registry.local:9001/docker.io/starlingx/ceph-config-helper:v1.15.0 download failed from local registry: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80"): APIError: 500 Server Error: Internal Server Error ("Get https://registry.local:9001/v2/docker.io/starlingx/ceph-config-helper/manifests/v1.15.0: Get https://192.168.49.80:9002/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fceph-config-helper%3Apull&service=192.168.204.1%3A9001: x509: certificate is valid for 192.168.204.1, 172.28.235.202, not 192.168.49.80") sysinv 2020-05-27 16:12:49.196 90332 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: KubeAppApplyFailure: Deployment of application platform-integ-apps (1.0-8) failed: failed to download one or more image(s). sysinv 2020-05-27 16:12:49.197 90332 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app platform-integ-apps sysinv 2020-05-27 16:13:07.152 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-13-07 patch sysinv 2020-05-27 16:13:07.152 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:14:47.810 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:14:47.811 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:14:47.812 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:14:47.813 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:14:47.813 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:15:12.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:12.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:12.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:12.791 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:12.791 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['subfunction_oper', 'subfunction_avail'] sysinv 2020-05-27 16:15:12.811 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:12.853 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:12.854 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:12.909 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-12 patch sysinv 2020-05-27 16:15:22.669 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-22 patch sysinv 2020-05-27 16:15:22.670 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:22.712 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-22 patch sysinv 2020-05-27 16:15:22.713 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:26.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-26 patch sysinv 2020-05-27 16:15:26.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:26.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-26 patch sysinv 2020-05-27 16:15:26.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-26 patch sysinv 2020-05-27 16:15:26.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:26.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-26 patch sysinv 2020-05-27 16:15:26.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:26.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-26 patch sysinv 2020-05-27 16:15:32.686 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-32 patch sysinv 2020-05-27 16:15:32.686 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:32.733 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-32 patch sysinv 2020-05-27 16:15:32.734 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:42.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-42 patch sysinv 2020-05-27 16:15:42.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:42.763 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-42 patch sysinv 2020-05-27 16:15:42.763 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:47.234 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-47 patch sysinv 2020-05-27 16:15:47.234 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:47.253 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-47 patch sysinv 2020-05-27 16:15:47.306 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-47 patch sysinv 2020-05-27 16:15:47.306 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:47.355 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-47 patch sysinv 2020-05-27 16:15:47.355 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:47.413 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-47 patch sysinv 2020-05-27 16:15:47.775 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:15:47.776 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:15:47.776 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:15:47.777 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:15:47.777 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:15:52.728 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-52 patch sysinv 2020-05-27 16:15:52.729 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:52.774 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-52 patch sysinv 2020-05-27 16:15:52.774 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:54.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-54 patch sysinv 2020-05-27 16:15:54.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:54.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-54 patch sysinv 2020-05-27 16:15:54.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-54 patch sysinv 2020-05-27 16:15:54.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:15:54.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-15-54 patch sysinv 2020-05-27 16:15:54.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:15:54.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-15-54 patch sysinv 2020-05-27 16:16:01.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-01 patch sysinv 2020-05-27 16:16:01.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:01.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-01 patch sysinv 2020-05-27 16:16:01.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-01 patch sysinv 2020-05-27 16:16:01.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:01.832 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-01 patch sysinv 2020-05-27 16:16:01.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:01.887 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-01 patch sysinv 2020-05-27 16:16:02.753 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-02 patch sysinv 2020-05-27 16:16:02.754 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:02.799 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-02 patch sysinv 2020-05-27 16:16:02.799 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:04.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-04 patch sysinv 2020-05-27 16:16:04.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:04.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-04 patch sysinv 2020-05-27 16:16:04.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-04 patch sysinv 2020-05-27 16:16:04.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:04.841 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-04 patch sysinv 2020-05-27 16:16:04.842 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:04.904 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-04 patch sysinv 2020-05-27 16:16:12.770 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-12 patch sysinv 2020-05-27 16:16:12.770 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:12.825 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-12 patch sysinv 2020-05-27 16:16:12.825 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:22.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:22.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.776 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.776 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:22.828 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.828 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:22.903 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.993 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-22 patch sysinv 2020-05-27 16:16:22.993 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:23.039 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-23 patch sysinv 2020-05-27 16:16:23.040 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:32.811 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-32 patch sysinv 2020-05-27 16:16:32.811 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:32.864 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-32 patch sysinv 2020-05-27 16:16:32.864 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:36.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-36 patch sysinv 2020-05-27 16:16:36.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:36.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-36 patch sysinv 2020-05-27 16:16:36.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-36 patch sysinv 2020-05-27 16:16:36.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:36.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-36 patch sysinv 2020-05-27 16:16:36.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:36.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-36 patch sysinv 2020-05-27 16:16:37.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-37 patch sysinv 2020-05-27 16:16:37.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:37.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-37 patch sysinv 2020-05-27 16:16:37.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-37 patch sysinv 2020-05-27 16:16:37.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:37.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-37 patch sysinv 2020-05-27 16:16:37.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:37.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-37 patch sysinv 2020-05-27 16:16:42.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-42 patch sysinv 2020-05-27 16:16:42.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:42.887 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-42 patch sysinv 2020-05-27 16:16:42.887 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:46.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-46 patch sysinv 2020-05-27 16:16:46.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:46.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-46 patch sysinv 2020-05-27 16:16:46.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-46 patch sysinv 2020-05-27 16:16:46.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:46.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-46 patch sysinv 2020-05-27 16:16:46.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:46.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-46 patch sysinv 2020-05-27 16:16:47.857 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:16:47.858 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:16:47.859 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:16:47.859 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:16:47.860 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:16:52.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-52 patch sysinv 2020-05-27 16:16:52.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:52.903 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-52 patch sysinv 2020-05-27 16:16:52.904 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:57.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-57 patch sysinv 2020-05-27 16:16:57.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:57.234 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-57 patch sysinv 2020-05-27 16:16:57.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-57 patch sysinv 2020-05-27 16:16:57.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:16:57.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-16-57 patch sysinv 2020-05-27 16:16:57.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:16:57.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-16-57 patch sysinv 2020-05-27 16:17:02.874 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-02 patch sysinv 2020-05-27 16:17:02.875 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:02.927 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-02 patch sysinv 2020-05-27 16:17:02.928 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:12.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:12.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:12.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:12.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:12.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:12.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:12.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:12.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:13.093 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-12 patch sysinv 2020-05-27 16:17:13.093 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:13.145 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-13 patch sysinv 2020-05-27 16:17:13.145 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:16.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-16 patch sysinv 2020-05-27 16:17:16.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:16.741 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-16 patch sysinv 2020-05-27 16:17:16.792 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-16 patch sysinv 2020-05-27 16:17:16.793 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:16.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-16 patch sysinv 2020-05-27 16:17:16.838 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:16.902 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-16 patch sysinv 2020-05-27 16:17:22.919 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-22 patch sysinv 2020-05-27 16:17:22.919 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:22.969 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-22 patch sysinv 2020-05-27 16:17:22.969 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:26.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-26 patch sysinv 2020-05-27 16:17:26.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:26.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-26 patch sysinv 2020-05-27 16:17:26.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-26 patch sysinv 2020-05-27 16:17:26.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:26.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-26 patch sysinv 2020-05-27 16:17:26.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:26.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-26 patch sysinv 2020-05-27 16:17:32.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:32.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:32.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:32.401 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.936 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.936 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:32.990 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-32 patch sysinv 2020-05-27 16:17:32.991 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:43.140 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-43 patch sysinv 2020-05-27 16:17:43.141 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:43.190 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-43 patch sysinv 2020-05-27 16:17:43.190 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:47.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-47 patch sysinv 2020-05-27 16:17:47.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:47.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-47 patch sysinv 2020-05-27 16:17:47.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-47 patch sysinv 2020-05-27 16:17:47.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:47.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-47 patch sysinv 2020-05-27 16:17:47.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:47.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-47 patch sysinv 2020-05-27 16:17:47.840 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:17:47.841 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:17:47.842 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:17:47.842 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:17:47.842 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:17:52.997 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-52 patch sysinv 2020-05-27 16:17:52.997 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:53.055 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-53 patch sysinv 2020-05-27 16:17:53.055 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:57.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-57 patch sysinv 2020-05-27 16:17:57.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:57.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-57 patch sysinv 2020-05-27 16:17:57.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-57 patch sysinv 2020-05-27 16:17:57.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:17:57.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-17-57 patch sysinv 2020-05-27 16:17:57.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:17:57.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-17-57 patch sysinv 2020-05-27 16:18:02.999 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-02 patch sysinv 2020-05-27 16:18:02.999 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:03.056 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-03 patch sysinv 2020-05-27 16:18:03.056 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:08.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-08 patch sysinv 2020-05-27 16:18:08.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:08.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-08 patch sysinv 2020-05-27 16:18:08.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-08 patch sysinv 2020-05-27 16:18:08.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:08.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-08 patch sysinv 2020-05-27 16:18:08.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:08.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-08 patch sysinv 2020-05-27 16:18:13.020 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-13 patch sysinv 2020-05-27 16:18:13.020 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:13.073 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-13 patch sysinv 2020-05-27 16:18:13.074 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:22.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-22 patch sysinv 2020-05-27 16:18:22.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:22.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-22 patch sysinv 2020-05-27 16:18:22.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-22 patch sysinv 2020-05-27 16:18:22.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:22.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-22 patch sysinv 2020-05-27 16:18:22.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:22.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-22 patch sysinv 2020-05-27 16:18:23.041 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-23 patch sysinv 2020-05-27 16:18:23.041 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:23.095 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-23 patch sysinv 2020-05-27 16:18:23.096 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:33.063 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-33 patch sysinv 2020-05-27 16:18:33.063 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:33.111 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-33 patch sysinv 2020-05-27 16:18:33.111 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:38.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-38 patch sysinv 2020-05-27 16:18:38.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:38.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-38 patch sysinv 2020-05-27 16:18:38.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-38 patch sysinv 2020-05-27 16:18:38.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:38.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-38 patch sysinv 2020-05-27 16:18:38.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:38.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-38 patch sysinv 2020-05-27 16:18:42.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-42 patch sysinv 2020-05-27 16:18:42.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:42.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-42 patch sysinv 2020-05-27 16:18:42.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-42 patch sysinv 2020-05-27 16:18:42.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:42.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-42 patch sysinv 2020-05-27 16:18:42.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:42.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-42 patch sysinv 2020-05-27 16:18:43.084 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-43 patch sysinv 2020-05-27 16:18:43.085 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:43.142 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-43 patch sysinv 2020-05-27 16:18:43.142 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:47.675 90332 INFO sysinv.conductor.manager [-] Reset managed application platform-integ-apps status to uploaded sysinv 2020-05-27 16:18:47.691 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'fsid', 'format': 'text'} sysinv 2020-05-27 16:18:47.699 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'3bedf8ca-b284-4cc3-abb8-8d1f547f2751\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'139940928235152'} sysinv 2020-05-27 16:18:47.704 90332 INFO ceph_client [-] Request params: url=https://controller-0:5001/request?wait=1, json={'prefix': 'status', 'format': 'json'} sysinv 2020-05-27 16:18:47.710 90332 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'{"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","health":{"checks":{},"status":"HEALTH_OK","overall_status":"HEALTH_WARN"},"election_epoch":25,"quorum":[0],"quorum_names":["controller-0"],"monmap":{"epoch":1,"fsid":"3bedf8ca-b284-4cc3-abb8-8d1f547f2751","modified":"2020-04-21 10:43:03.193656","created":"2020-04-21 10:43:03.193656","features":{"persistent":["kraken","luminous","mimic","osdmap-prune"],"optional":[]},"mons":[{"rank":0,"name":"controller-0","addr":"192.168.204.2:6789/0","public_addr":"192.168.204.2:6789/0"}]},"osdmap":{"osdmap":{"epoch":69,"num_osds":1,"num_up_osds":1,"num_in_osds":1,"full":false,"nearfull":false,"num_remapped_pgs":0}},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":600}],"num_pgs":600,"num_pools":5,"num_objects":515,"data_bytes":1661762291,"bytes_used":1764102144,"bytes_avail":1596701519872,"bytes_total":1598465622016},"fsmap":{"epoch":1,"by_rank":[]},"mgrmap":{"epoch":79,"active_gid":904130,"active_name":"controller-0","active_addr":"192.168.204.2:6804/90207","available":true,"standbys":[],"modules":["restful"],"available_modules":[{"name":"balancer","can_run":true,"error_string":""},{"name":"dashboard","can_run":false,"error_string":"Frontend assets not found: incomplete build?"},{"name":"hello","can_run":true,"error_string":""},{"name":"iostat","can_run":true,"error_string":""},{"name":"localpool","can_run":true,"error_string":""},{"name":"prometheus","can_run":true,"error_string":""},{"name":"restful","can_run":true,"error_string":""},{"name":"selftest","can_run":true,"error_string":""},{"name":"smart","can_run":false,"error_string":""},{"name":"status","can_run":true,"error_string":""},{"name":"telegraf","can_run":true,"error_string":""},{"name":"telemetry","can_run":true,"error_string":""},{"name":"zabbix","can_run":true,"error_string":""}],"services":{"restful":"https://controller-0:5001/"}},"servicemap":{"epoch":1,"modified":"0.000000","services":{}}}\n', u'outs': u'', u'command': u'status format=json'}], u'is_finished': True, u'id': u'139940928057744'} sysinv 2020-05-27 16:18:48.058 90332 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://192.168.204.1:5491/v1/query/ hdr:None payload:None sysinv 2020-05-27 16:18:48.126 90332 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}} sysinv 2020-05-27 16:18:48.136 90332 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying... sysinv 2020-05-27 16:18:48.295 90332 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app platform-integ-apps sysinv 2020-05-27 16:18:48.597 90332 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) apply started. sysinv 2020-05-27 16:18:48.621 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.622 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.623 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/namespaces/kube-system/secrets/default-registry-key: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.623 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_secret: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app [-] HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1947, in perform_app_apply 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app self._create_local_registry_secrets(app.name) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1002, in _create_local_registry_secrets 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app self._kube.kube_get_secret(DOCKER_REGISTRY_SECRET, ns)): 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 194, in kube_get_secret 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app c.read_namespaced_secret(name, namespace) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 19486, in read_namespaced_secret 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app (data) = self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 19577, in read_namespaced_secret_with_http_info 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app collection_formats=collection_formats) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app _request_timeout=_request_timeout) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app headers=headers) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app query_params=query_params) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app headers=headers) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app **urlopen_kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app return self.urlopen(method, url, **extra_kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app **response_kw) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app _stacktrace=sys.exc_info()[2]) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:18:48.624 90332 ERROR sysinv.conductor.kube_app sysinv 2020-05-27 16:18:48.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-48 patch sysinv 2020-05-27 16:18:48.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:48.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-48 patch sysinv 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (psycopg2.DataError) value too long for type character varying(255) [SQL: 'UPDATE kube_app SET updated_at=%(updated_at)s, status=%(status)s, progress=%(progress)s WHERE kube_app.id = %(id_1)s'] [parameters: {'status': u'apply-failed', 'progress': u"HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/kube-system/secrets/default-registry-key (Ca ... (16 characters truncated) ... ctionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',))", 'id_1': 1, 'updated_at': datetime.datetime(2020, 5, 27, 16, 18, 48, 780916)}]: DataError: value too long for type character varying(255) 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters context) 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters DataError: value too long for type character varying(255) 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters 2020-05-27 16:18:48.781 90332 ERROR oslo_db.sqlalchemy.exc_filters sysinv 2020-05-27 16:18:48.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-48 patch sysinv 2020-05-27 16:18:48.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:48.806 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.807 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.807 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:18:48.808 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:18:48.808 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:18:48.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-48 patch sysinv 2020-05-27 16:18:48.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:48.999 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-48 patch sysinv 2020-05-27 16:18:53.123 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-53 patch sysinv 2020-05-27 16:18:53.124 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:53.182 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-53 patch sysinv 2020-05-27 16:18:53.183 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:55.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-55 patch sysinv 2020-05-27 16:18:55.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:55.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-55 patch sysinv 2020-05-27 16:18:55.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-55 patch sysinv 2020-05-27 16:18:55.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:18:55.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-18-55 patch sysinv 2020-05-27 16:18:55.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:18:55.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-18-55 patch sysinv 2020-05-27 16:19:03.135 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-03 patch sysinv 2020-05-27 16:19:03.135 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:03.185 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-03 patch sysinv 2020-05-27 16:19:03.186 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:13.147 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-13 patch sysinv 2020-05-27 16:19:13.148 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:13.200 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-13 patch sysinv 2020-05-27 16:19:13.201 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:18.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:18.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.288 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.289 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:18.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:18.395 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:18.744 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.790 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.790 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:18.841 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:18.842 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:18.897 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-18 patch sysinv 2020-05-27 16:19:23.178 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-23 patch sysinv 2020-05-27 16:19:23.179 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:23.236 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-23 patch sysinv 2020-05-27 16:19:23.236 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:28.725 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-28 patch sysinv 2020-05-27 16:19:28.725 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:28.744 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-28 patch sysinv 2020-05-27 16:19:28.794 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-28 patch sysinv 2020-05-27 16:19:28.794 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:28.842 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-28 patch sysinv 2020-05-27 16:19:28.842 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:28.897 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-28 patch sysinv 2020-05-27 16:19:29.213 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-29 patch sysinv 2020-05-27 16:19:29.213 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:29.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-29 patch sysinv 2020-05-27 16:19:29.278 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-29 patch sysinv 2020-05-27 16:19:29.278 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:29.329 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-29 patch sysinv 2020-05-27 16:19:29.329 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:29.384 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-29 patch sysinv 2020-05-27 16:19:33.190 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-33 patch sysinv 2020-05-27 16:19:33.191 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:33.240 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-33 patch sysinv 2020-05-27 16:19:33.240 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:43.209 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-43 patch sysinv 2020-05-27 16:19:43.210 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:43.258 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-43 patch sysinv 2020-05-27 16:19:43.259 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:51.713 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-51 patch sysinv 2020-05-27 16:19:51.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:51.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-51 patch sysinv 2020-05-27 16:19:51.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-51 patch sysinv 2020-05-27 16:19:51.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:51.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-51 patch sysinv 2020-05-27 16:19:51.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:19:51.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-19-51 patch sysinv 2020-05-27 16:19:53.249 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-53 patch sysinv 2020-05-27 16:19:53.249 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:19:53.300 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-19-53 patch sysinv 2020-05-27 16:19:53.301 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:00.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-00 patch sysinv 2020-05-27 16:20:00.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:00.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-00 patch sysinv 2020-05-27 16:20:00.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-00 patch sysinv 2020-05-27 16:20:00.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:00.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-00 patch sysinv 2020-05-27 16:20:00.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:00.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-00 patch sysinv 2020-05-27 16:20:03.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:03.240 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:03.331 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.331 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:03.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.472 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.472 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:03.657 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-03 patch sysinv 2020-05-27 16:20:03.657 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:10.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-10 patch sysinv 2020-05-27 16:20:10.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:10.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-10 patch sysinv 2020-05-27 16:20:10.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-10 patch sysinv 2020-05-27 16:20:10.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:10.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-10 patch sysinv 2020-05-27 16:20:10.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:10.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-10 patch sysinv 2020-05-27 16:20:13.275 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-13 patch sysinv 2020-05-27 16:20:13.275 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:13.318 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-13 patch sysinv 2020-05-27 16:20:13.318 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:23.296 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-23 patch sysinv 2020-05-27 16:20:23.297 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:23.339 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-23 patch sysinv 2020-05-27 16:20:23.340 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:25.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-25 patch sysinv 2020-05-27 16:20:25.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:25.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-25 patch sysinv 2020-05-27 16:20:25.281 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-25 patch sysinv 2020-05-27 16:20:25.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:25.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-25 patch sysinv 2020-05-27 16:20:25.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:25.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-25 patch sysinv 2020-05-27 16:20:33.315 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-33 patch sysinv 2020-05-27 16:20:33.316 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:33.363 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-33 patch sysinv 2020-05-27 16:20:33.364 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:37.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-37 patch sysinv 2020-05-27 16:20:37.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:37.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-37 patch sysinv 2020-05-27 16:20:37.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-37 patch sysinv 2020-05-27 16:20:37.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:37.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-37 patch sysinv 2020-05-27 16:20:37.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:37.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-37 patch sysinv 2020-05-27 16:20:42.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-42 patch sysinv 2020-05-27 16:20:42.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:42.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-42 patch sysinv 2020-05-27 16:20:42.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-42 patch sysinv 2020-05-27 16:20:42.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:42.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-42 patch sysinv 2020-05-27 16:20:42.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:42.396 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-42 patch sysinv 2020-05-27 16:20:43.339 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-43 patch sysinv 2020-05-27 16:20:43.340 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:43.390 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-43 patch sysinv 2020-05-27 16:20:43.390 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:47.990 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:20:47.991 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:20:47.992 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:20:47.992 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:20:47.993 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:20:52.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-52 patch sysinv 2020-05-27 16:20:52.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:52.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-52 patch sysinv 2020-05-27 16:20:52.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-52 patch sysinv 2020-05-27 16:20:52.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:52.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-52 patch sysinv 2020-05-27 16:20:52.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:20:52.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-20-52 patch sysinv 2020-05-27 16:20:53.376 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-53 patch sysinv 2020-05-27 16:20:53.376 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:20:53.425 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-20-53 patch sysinv 2020-05-27 16:20:53.425 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:00.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-00 patch sysinv 2020-05-27 16:21:00.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:00.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-00 patch sysinv 2020-05-27 16:21:00.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-00 patch sysinv 2020-05-27 16:21:00.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:00.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-00 patch sysinv 2020-05-27 16:21:00.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:00.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-00 patch sysinv 2020-05-27 16:21:03.374 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-03 patch sysinv 2020-05-27 16:21:03.375 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:03.423 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-03 patch sysinv 2020-05-27 16:21:03.423 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:12.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-12 patch sysinv 2020-05-27 16:21:12.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:12.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-12 patch sysinv 2020-05-27 16:21:12.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-12 patch sysinv 2020-05-27 16:21:12.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:12.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-12 patch sysinv 2020-05-27 16:21:12.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:12.395 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-12 patch sysinv 2020-05-27 16:21:13.395 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-13 patch sysinv 2020-05-27 16:21:13.396 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:13.450 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-13 patch sysinv 2020-05-27 16:21:13.450 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:23.421 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-23 patch sysinv 2020-05-27 16:21:23.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:23.471 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-23 patch sysinv 2020-05-27 16:21:23.472 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:24.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-24 patch sysinv 2020-05-27 16:21:24.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:24.241 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-24 patch sysinv 2020-05-27 16:21:24.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-24 patch sysinv 2020-05-27 16:21:24.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:24.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-24 patch sysinv 2020-05-27 16:21:24.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:24.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-24 patch sysinv 2020-05-27 16:21:33.442 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-33 patch sysinv 2020-05-27 16:21:33.442 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:33.494 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-33 patch sysinv 2020-05-27 16:21:33.495 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:34.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:34.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:34.339 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.339 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:34.398 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.713 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:34.732 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:34.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:34.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:34.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-34 patch sysinv 2020-05-27 16:21:43.462 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-43 patch sysinv 2020-05-27 16:21:43.462 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:43.508 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-43 patch sysinv 2020-05-27 16:21:43.509 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:47.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-47 patch sysinv 2020-05-27 16:21:47.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:47.348 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-47 patch sysinv 2020-05-27 16:21:47.383 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-47 patch sysinv 2020-05-27 16:21:47.384 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:47.433 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-47 patch sysinv 2020-05-27 16:21:47.434 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:21:47.488 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-21-47 patch sysinv 2020-05-27 16:21:48.072 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:21:48.072 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:21:48.073 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:21:48.074 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:21:48.074 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:21:53.504 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-53 patch sysinv 2020-05-27 16:21:53.505 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:21:53.549 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-21-53 patch sysinv 2020-05-27 16:21:53.549 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:03.506 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-03 patch sysinv 2020-05-27 16:22:03.506 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:03.554 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-03 patch sysinv 2020-05-27 16:22:03.555 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:05.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-05 patch sysinv 2020-05-27 16:22:05.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:05.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-05 patch sysinv 2020-05-27 16:22:05.793 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-05 patch sysinv 2020-05-27 16:22:05.793 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:05.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-05 patch sysinv 2020-05-27 16:22:05.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:05.893 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-05 patch sysinv 2020-05-27 16:22:09.302 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-09 patch sysinv 2020-05-27 16:22:09.303 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:09.322 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-09 patch sysinv 2020-05-27 16:22:09.369 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-09 patch sysinv 2020-05-27 16:22:09.369 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:09.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-09 patch sysinv 2020-05-27 16:22:09.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:09.477 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-09 patch sysinv 2020-05-27 16:22:13.527 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-13 patch sysinv 2020-05-27 16:22:13.527 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:13.580 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-13 patch sysinv 2020-05-27 16:22:13.581 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:15.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-15 patch sysinv 2020-05-27 16:22:15.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:15.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-15 patch sysinv 2020-05-27 16:22:15.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-15 patch sysinv 2020-05-27 16:22:15.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:15.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-15 patch sysinv 2020-05-27 16:22:15.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:15.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-15 patch sysinv 2020-05-27 16:22:21.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-21 patch sysinv 2020-05-27 16:22:21.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:21.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-21 patch sysinv 2020-05-27 16:22:21.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-21 patch sysinv 2020-05-27 16:22:21.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:21.345 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-21 patch sysinv 2020-05-27 16:22:21.345 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:21.429 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-21 patch sysinv 2020-05-27 16:22:23.546 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-23 patch sysinv 2020-05-27 16:22:23.546 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:23.595 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-23 patch sysinv 2020-05-27 16:22:23.596 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:33.568 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-33 patch sysinv 2020-05-27 16:22:33.569 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:33.613 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-33 patch sysinv 2020-05-27 16:22:33.613 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:43.650 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.651 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:43.692 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.693 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:43.740 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.741 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:43.760 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.792 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.793 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:43.844 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:43.845 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:43.902 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-43 patch sysinv 2020-05-27 16:22:46.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-46 patch sysinv 2020-05-27 16:22:46.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:46.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-46 patch sysinv 2020-05-27 16:22:46.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-46 patch sysinv 2020-05-27 16:22:46.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:46.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-46 patch sysinv 2020-05-27 16:22:46.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:46.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-46 patch sysinv 2020-05-27 16:22:48.151 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:22:48.152 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:22:48.153 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:22:48.154 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:22:48.154 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:22:53.625 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-53 patch sysinv 2020-05-27 16:22:53.626 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:53.672 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-53 patch sysinv 2020-05-27 16:22:53.672 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:56.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-56 patch sysinv 2020-05-27 16:22:56.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:56.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-56 patch sysinv 2020-05-27 16:22:56.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-56 patch sysinv 2020-05-27 16:22:56.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:56.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-56 patch sysinv 2020-05-27 16:22:56.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:56.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-56 patch sysinv 2020-05-27 16:22:57.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-57 patch sysinv 2020-05-27 16:22:57.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:57.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-57 patch sysinv 2020-05-27 16:22:57.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-57 patch sysinv 2020-05-27 16:22:57.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:22:57.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-22-57 patch sysinv 2020-05-27 16:22:57.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:22:57.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-22-57 patch sysinv 2020-05-27 16:23:03.623 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-03 patch sysinv 2020-05-27 16:23:03.624 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:03.678 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-03 patch sysinv 2020-05-27 16:23:03.678 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:13.645 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-13 patch sysinv 2020-05-27 16:23:13.646 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:13.693 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-13 patch sysinv 2020-05-27 16:23:13.694 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:18.823 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-18 patch sysinv 2020-05-27 16:23:18.824 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:18.848 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-18 patch sysinv 2020-05-27 16:23:18.904 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-18 patch sysinv 2020-05-27 16:23:18.904 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:18.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-18 patch sysinv 2020-05-27 16:23:18.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:19.010 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-19 patch sysinv 2020-05-27 16:23:23.671 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-23 patch sysinv 2020-05-27 16:23:23.671 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:23.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-23 patch sysinv 2020-05-27 16:23:23.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:28.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-28 patch sysinv 2020-05-27 16:23:28.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:28.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-28 patch sysinv 2020-05-27 16:23:28.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-28 patch sysinv 2020-05-27 16:23:28.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:28.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-28 patch sysinv 2020-05-27 16:23:28.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:28.896 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-28 patch sysinv 2020-05-27 16:23:31.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-31 patch sysinv 2020-05-27 16:23:31.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:31.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-31 patch sysinv 2020-05-27 16:23:31.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-31 patch sysinv 2020-05-27 16:23:31.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:31.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-31 patch sysinv 2020-05-27 16:23:31.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:31.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-31 patch sysinv 2020-05-27 16:23:33.695 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-33 patch sysinv 2020-05-27 16:23:33.695 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:33.735 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-33 patch sysinv 2020-05-27 16:23:33.735 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:39.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-39 patch sysinv 2020-05-27 16:23:39.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:39.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-39 patch sysinv 2020-05-27 16:23:39.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-39 patch sysinv 2020-05-27 16:23:39.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:39.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-39 patch sysinv 2020-05-27 16:23:39.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:39.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-39 patch sysinv 2020-05-27 16:23:43.711 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-43 patch sysinv 2020-05-27 16:23:43.712 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:43.760 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-43 patch sysinv 2020-05-27 16:23:43.761 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:48.336 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:23:48.337 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:23:48.338 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:23:48.338 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:23:48.339 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:23:52.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-52 patch sysinv 2020-05-27 16:23:52.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:52.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-52 patch sysinv 2020-05-27 16:23:52.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-52 patch sysinv 2020-05-27 16:23:52.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:52.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-52 patch sysinv 2020-05-27 16:23:52.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:23:52.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-23-52 patch sysinv 2020-05-27 16:23:53.747 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-53 patch sysinv 2020-05-27 16:23:53.748 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:23:53.796 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-23-53 patch sysinv 2020-05-27 16:23:53.796 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:03.749 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-03 patch sysinv 2020-05-27 16:24:03.750 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:03.798 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-03 patch sysinv 2020-05-27 16:24:03.798 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:06.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-06 patch sysinv 2020-05-27 16:24:06.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:06.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-06 patch sysinv 2020-05-27 16:24:06.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-06 patch sysinv 2020-05-27 16:24:06.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:06.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-06 patch sysinv 2020-05-27 16:24:06.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:06.894 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-06 patch sysinv 2020-05-27 16:24:10.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-10 patch sysinv 2020-05-27 16:24:10.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:10.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-10 patch sysinv 2020-05-27 16:24:10.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-10 patch sysinv 2020-05-27 16:24:10.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:10.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-10 patch sysinv 2020-05-27 16:24:10.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:10.396 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-10 patch sysinv 2020-05-27 16:24:13.770 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-13 patch sysinv 2020-05-27 16:24:13.770 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:13.817 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-13 patch sysinv 2020-05-27 16:24:13.817 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:20.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-20 patch sysinv 2020-05-27 16:24:20.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:20.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-20 patch sysinv 2020-05-27 16:24:20.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-20 patch sysinv 2020-05-27 16:24:20.286 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:20.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-20 patch sysinv 2020-05-27 16:24:20.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:20.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-20 patch sysinv 2020-05-27 16:24:23.794 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-23 patch sysinv 2020-05-27 16:24:23.794 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:23.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-23 patch sysinv 2020-05-27 16:24:23.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:27.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-27 patch sysinv 2020-05-27 16:24:27.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:27.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-27 patch sysinv 2020-05-27 16:24:27.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-27 patch sysinv 2020-05-27 16:24:27.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:27.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-27 patch sysinv 2020-05-27 16:24:27.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:27.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-27 patch sysinv 2020-05-27 16:24:33.812 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-33 patch sysinv 2020-05-27 16:24:33.813 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:33.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-33 patch sysinv 2020-05-27 16:24:33.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:41.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-41 patch sysinv 2020-05-27 16:24:41.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:41.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-41 patch sysinv 2020-05-27 16:24:41.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-41 patch sysinv 2020-05-27 16:24:41.788 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:41.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-41 patch sysinv 2020-05-27 16:24:41.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:42.013 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-42 patch sysinv 2020-05-27 16:24:43.831 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-43 patch sysinv 2020-05-27 16:24:43.831 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:43.879 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-43 patch sysinv 2020-05-27 16:24:43.879 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:48.210 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:24:48.210 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:24:48.211 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:24:48.212 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:24:48.212 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:24:51.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-51 patch sysinv 2020-05-27 16:24:51.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:51.243 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-51 patch sysinv 2020-05-27 16:24:51.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-51 patch sysinv 2020-05-27 16:24:51.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:51.346 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-51 patch sysinv 2020-05-27 16:24:51.346 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:24:51.439 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-24-51 patch sysinv 2020-05-27 16:24:53.864 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-53 patch sysinv 2020-05-27 16:24:53.865 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:24:53.902 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-24-53 patch sysinv 2020-05-27 16:24:53.902 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:01.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-01 patch sysinv 2020-05-27 16:25:01.721 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:01.742 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-01 patch sysinv 2020-05-27 16:25:01.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-01 patch sysinv 2020-05-27 16:25:01.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:01.839 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-01 patch sysinv 2020-05-27 16:25:01.840 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:01.904 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-01 patch sysinv 2020-05-27 16:25:02.213 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-02 patch sysinv 2020-05-27 16:25:02.213 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:02.231 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-02 patch sysinv 2020-05-27 16:25:02.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-02 patch sysinv 2020-05-27 16:25:02.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:02.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-02 patch sysinv 2020-05-27 16:25:02.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:02.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-02 patch sysinv 2020-05-27 16:25:03.871 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-03 patch sysinv 2020-05-27 16:25:03.872 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:03.922 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-03 patch sysinv 2020-05-27 16:25:03.923 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:13.891 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-13 patch sysinv 2020-05-27 16:25:13.892 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:13.943 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-13 patch sysinv 2020-05-27 16:25:13.943 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:16.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-16 patch sysinv 2020-05-27 16:25:16.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:16.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-16 patch sysinv 2020-05-27 16:25:16.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-16 patch sysinv 2020-05-27 16:25:16.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:16.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-16 patch sysinv 2020-05-27 16:25:16.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:16.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-16 patch sysinv 2020-05-27 16:25:23.914 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-23 patch sysinv 2020-05-27 16:25:23.915 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:23.964 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-23 patch sysinv 2020-05-27 16:25:23.964 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:32.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-32 patch sysinv 2020-05-27 16:25:32.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:32.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-32 patch sysinv 2020-05-27 16:25:32.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-32 patch sysinv 2020-05-27 16:25:32.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:32.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-32 patch sysinv 2020-05-27 16:25:32.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:32.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-32 patch sysinv 2020-05-27 16:25:33.934 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-33 patch sysinv 2020-05-27 16:25:33.934 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:33.982 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-33 patch sysinv 2020-05-27 16:25:33.983 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:37.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-37 patch sysinv 2020-05-27 16:25:37.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:37.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-37 patch sysinv 2020-05-27 16:25:37.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-37 patch sysinv 2020-05-27 16:25:37.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:37.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-37 patch sysinv 2020-05-27 16:25:37.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:37.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-37 patch sysinv 2020-05-27 16:25:43.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:43.234 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:43.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:43.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:43.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:44.006 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-43 patch sysinv 2020-05-27 16:25:44.006 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:48.296 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:25:48.297 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:25:48.298 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:25:48.298 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:25:48.299 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:25:50.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-50 patch sysinv 2020-05-27 16:25:50.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:50.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-50 patch sysinv 2020-05-27 16:25:50.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-50 patch sysinv 2020-05-27 16:25:50.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:50.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-50 patch sysinv 2020-05-27 16:25:50.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:25:50.393 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-25-50 patch sysinv 2020-05-27 16:25:53.990 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-53 patch sysinv 2020-05-27 16:25:53.991 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:25:54.025 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-25-54 patch sysinv 2020-05-27 16:25:54.025 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:03.996 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-03 patch sysinv 2020-05-27 16:26:03.997 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:04.051 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-04 patch sysinv 2020-05-27 16:26:04.051 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:11.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-11 patch sysinv 2020-05-27 16:26:11.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:11.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-11 patch sysinv 2020-05-27 16:26:11.888 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-11 patch sysinv 2020-05-27 16:26:11.889 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:11.934 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-11 patch sysinv 2020-05-27 16:26:11.934 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:11.990 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-11 patch sysinv 2020-05-27 16:26:14.020 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.020 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:14.074 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.074 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:14.301 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.301 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:14.319 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.372 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.372 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:14.421 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:14.421 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:14.478 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-14 patch sysinv 2020-05-27 16:26:24.038 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.039 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:24.091 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.091 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:24.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:24.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.273 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.274 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:24.326 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.326 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:24.382 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.450 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.451 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:24.468 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.519 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.519 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:24.570 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:24.570 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:24.624 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-24 patch sysinv 2020-05-27 16:26:34.059 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-34 patch sysinv 2020-05-27 16:26:34.060 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:34.113 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-34 patch sysinv 2020-05-27 16:26:34.113 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:44.083 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-44 patch sysinv 2020-05-27 16:26:44.083 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:44.138 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-44 patch sysinv 2020-05-27 16:26:44.138 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:46.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-46 patch sysinv 2020-05-27 16:26:46.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:46.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-46 patch sysinv 2020-05-27 16:26:46.791 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-46 patch sysinv 2020-05-27 16:26:46.792 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:46.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-46 patch sysinv 2020-05-27 16:26:46.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:46.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-46 patch sysinv 2020-05-27 16:26:48.372 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:26:48.373 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:26:48.374 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:26:48.375 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:26:48.375 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:26:54.124 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.125 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:54.174 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.175 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:54.307 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.307 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:54.326 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.378 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.379 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:54.423 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:54.424 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:54.487 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-54 patch sysinv 2020-05-27 16:26:59.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-59 patch sysinv 2020-05-27 16:26:59.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:59.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-59 patch sysinv 2020-05-27 16:26:59.275 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-59 patch sysinv 2020-05-27 16:26:59.275 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:26:59.326 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-26-59 patch sysinv 2020-05-27 16:26:59.326 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:26:59.381 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-26-59 patch sysinv 2020-05-27 16:27:04.125 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.125 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:04.171 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.171 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:04.302 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.302 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:04.321 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.372 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.373 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:04.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:04.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:04.478 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-04 patch sysinv 2020-05-27 16:27:14.143 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-14 patch sysinv 2020-05-27 16:27:14.143 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:14.195 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-14 patch sysinv 2020-05-27 16:27:14.195 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:21.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-21 patch sysinv 2020-05-27 16:27:21.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:21.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-21 patch sysinv 2020-05-27 16:27:21.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-21 patch sysinv 2020-05-27 16:27:21.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:21.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-21 patch sysinv 2020-05-27 16:27:21.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:21.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-21 patch sysinv 2020-05-27 16:27:24.166 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-24 patch sysinv 2020-05-27 16:27:24.167 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:24.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-24 patch sysinv 2020-05-27 16:27:24.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:33.721 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-33 patch sysinv 2020-05-27 16:27:33.722 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:33.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-33 patch sysinv 2020-05-27 16:27:33.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-33 patch sysinv 2020-05-27 16:27:33.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:33.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-33 patch sysinv 2020-05-27 16:27:33.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:34.001 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.183 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.183 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:34.239 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.239 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:34.303 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.304 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:34.320 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.371 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.371 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:34.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:34.422 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:34.478 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-34 patch sysinv 2020-05-27 16:27:44.356 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.356 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:44.392 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.392 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:44.443 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.443 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:44.463 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.494 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.494 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:44.544 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:44.545 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:44.600 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-44 patch sysinv 2020-05-27 16:27:48.352 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:27:48.353 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:27:48.353 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:27:48.354 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:27:48.354 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:27:54.242 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-54 patch sysinv 2020-05-27 16:27:54.243 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:54.300 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-54 patch sysinv 2020-05-27 16:27:54.300 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:55.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-55 patch sysinv 2020-05-27 16:27:55.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:55.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-55 patch sysinv 2020-05-27 16:27:55.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-55 patch sysinv 2020-05-27 16:27:55.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:27:55.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-27-55 patch sysinv 2020-05-27 16:27:55.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:27:55.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-27-55 patch sysinv 2020-05-27 16:28:04.247 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-04 patch sysinv 2020-05-27 16:28:04.248 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:04.303 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-04 patch sysinv 2020-05-27 16:28:04.304 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:06.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-06 patch sysinv 2020-05-27 16:28:06.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:06.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-06 patch sysinv 2020-05-27 16:28:06.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-06 patch sysinv 2020-05-27 16:28:06.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:06.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-06 patch sysinv 2020-05-27 16:28:06.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:06.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-06 patch sysinv 2020-05-27 16:28:14.262 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-14 patch sysinv 2020-05-27 16:28:14.262 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:14.312 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-14 patch sysinv 2020-05-27 16:28:14.312 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:15.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-15 patch sysinv 2020-05-27 16:28:15.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:15.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-15 patch sysinv 2020-05-27 16:28:15.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-15 patch sysinv 2020-05-27 16:28:15.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:15.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-15 patch sysinv 2020-05-27 16:28:15.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:15.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-15 patch sysinv 2020-05-27 16:28:24.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-24 patch sysinv 2020-05-27 16:28:24.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:24.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-24 patch sysinv 2020-05-27 16:28:24.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:26.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-26 patch sysinv 2020-05-27 16:28:26.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:26.232 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-26 patch sysinv 2020-05-27 16:28:26.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-26 patch sysinv 2020-05-27 16:28:26.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:26.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-26 patch sysinv 2020-05-27 16:28:26.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:26.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-26 patch sysinv 2020-05-27 16:28:29.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-29 patch sysinv 2020-05-27 16:28:29.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:29.232 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-29 patch sysinv 2020-05-27 16:28:29.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-29 patch sysinv 2020-05-27 16:28:29.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:29.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-29 patch sysinv 2020-05-27 16:28:29.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:29.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-29 patch sysinv 2020-05-27 16:28:34.304 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-34 patch sysinv 2020-05-27 16:28:34.304 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:34.355 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-34 patch sysinv 2020-05-27 16:28:34.355 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:40.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-40 patch sysinv 2020-05-27 16:28:40.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:40.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-40 patch sysinv 2020-05-27 16:28:40.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-40 patch sysinv 2020-05-27 16:28:40.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:40.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-40 patch sysinv 2020-05-27 16:28:40.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:40.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-40 patch sysinv 2020-05-27 16:28:44.324 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-44 patch sysinv 2020-05-27 16:28:44.324 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:44.378 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-44 patch sysinv 2020-05-27 16:28:44.379 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:48.639 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:28:48.640 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:28:48.640 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:28:48.641 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:28:48.641 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:28:54.361 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-54 patch sysinv 2020-05-27 16:28:54.362 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:54.525 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-54 patch sysinv 2020-05-27 16:28:54.525 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:57.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-57 patch sysinv 2020-05-27 16:28:57.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:57.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-57 patch sysinv 2020-05-27 16:28:57.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-57 patch sysinv 2020-05-27 16:28:57.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:28:57.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-28-57 patch sysinv 2020-05-27 16:28:57.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:28:57.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-28-57 patch sysinv 2020-05-27 16:29:03.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-03 patch sysinv 2020-05-27 16:29:03.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:03.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-03 patch sysinv 2020-05-27 16:29:03.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-03 patch sysinv 2020-05-27 16:29:03.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:03.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-03 patch sysinv 2020-05-27 16:29:03.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:03.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-03 patch sysinv 2020-05-27 16:29:04.366 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-04 patch sysinv 2020-05-27 16:29:04.367 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:04.417 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-04 patch sysinv 2020-05-27 16:29:04.418 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:07.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-07 patch sysinv 2020-05-27 16:29:07.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:07.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-07 patch sysinv 2020-05-27 16:29:07.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-07 patch sysinv 2020-05-27 16:29:07.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:07.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-07 patch sysinv 2020-05-27 16:29:07.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:07.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-07 patch sysinv 2020-05-27 16:29:14.391 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.391 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:14.435 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.435 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:14.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:14.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:14.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:14.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:14.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-14 patch sysinv 2020-05-27 16:29:24.409 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-24 patch sysinv 2020-05-27 16:29:24.409 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:24.459 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-24 patch sysinv 2020-05-27 16:29:24.460 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:34.436 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-34 patch sysinv 2020-05-27 16:29:34.437 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:34.477 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-34 patch sysinv 2020-05-27 16:29:34.478 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:36.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-36 patch sysinv 2020-05-27 16:29:36.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:36.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-36 patch sysinv 2020-05-27 16:29:36.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-36 patch sysinv 2020-05-27 16:29:36.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:36.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-36 patch sysinv 2020-05-27 16:29:36.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:36.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-36 patch sysinv 2020-05-27 16:29:38.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-38 patch sysinv 2020-05-27 16:29:38.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:38.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-38 patch sysinv 2020-05-27 16:29:38.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-38 patch sysinv 2020-05-27 16:29:38.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:38.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-38 patch sysinv 2020-05-27 16:29:38.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:38.899 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-38 patch sysinv 2020-05-27 16:29:44.454 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-44 patch sysinv 2020-05-27 16:29:44.454 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:44.497 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-44 patch sysinv 2020-05-27 16:29:44.498 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:48.225 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-48 patch sysinv 2020-05-27 16:29:48.226 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:48.248 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-48 patch sysinv 2020-05-27 16:29:48.289 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-48 patch sysinv 2020-05-27 16:29:48.289 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:48.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-48 patch sysinv 2020-05-27 16:29:48.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:48.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-48 patch sysinv 2020-05-27 16:29:49.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-49 patch sysinv 2020-05-27 16:29:49.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:49.241 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-49 patch sysinv 2020-05-27 16:29:49.281 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-49 patch sysinv 2020-05-27 16:29:49.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:49.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-49 patch sysinv 2020-05-27 16:29:49.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:29:49.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-29-49 patch sysinv 2020-05-27 16:29:54.488 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-54 patch sysinv 2020-05-27 16:29:54.488 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:29:54.538 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-29-54 patch sysinv 2020-05-27 16:29:54.538 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:04.491 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-04 patch sysinv 2020-05-27 16:30:04.491 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:04.540 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-04 patch sysinv 2020-05-27 16:30:04.540 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:10.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-10 patch sysinv 2020-05-27 16:30:10.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:10.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-10 patch sysinv 2020-05-27 16:30:10.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-10 patch sysinv 2020-05-27 16:30:10.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:10.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-10 patch sysinv 2020-05-27 16:30:10.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:10.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-10 patch sysinv 2020-05-27 16:30:14.520 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-14 patch sysinv 2020-05-27 16:30:14.520 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:14.560 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-14 patch sysinv 2020-05-27 16:30:14.560 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:20.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-20 patch sysinv 2020-05-27 16:30:20.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:20.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-20 patch sysinv 2020-05-27 16:30:20.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-20 patch sysinv 2020-05-27 16:30:20.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:20.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-20 patch sysinv 2020-05-27 16:30:20.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:20.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-20 patch sysinv 2020-05-27 16:30:21.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-21 patch sysinv 2020-05-27 16:30:21.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:21.241 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-21 patch sysinv 2020-05-27 16:30:21.390 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-21 patch sysinv 2020-05-27 16:30:21.390 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:21.435 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-21 patch sysinv 2020-05-27 16:30:21.435 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:21.493 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-21 patch sysinv 2020-05-27 16:30:24.534 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-24 patch sysinv 2020-05-27 16:30:24.534 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:24.579 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-24 patch sysinv 2020-05-27 16:30:24.579 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:30.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-30 patch sysinv 2020-05-27 16:30:30.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:30.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-30 patch sysinv 2020-05-27 16:30:30.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-30 patch sysinv 2020-05-27 16:30:30.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:30.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-30 patch sysinv 2020-05-27 16:30:30.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:30.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-30 patch sysinv 2020-05-27 16:30:34.553 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-34 patch sysinv 2020-05-27 16:30:34.553 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:34.601 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-34 patch sysinv 2020-05-27 16:30:34.601 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:43.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-43 patch sysinv 2020-05-27 16:30:43.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:43.732 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-43 patch sysinv 2020-05-27 16:30:43.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-43 patch sysinv 2020-05-27 16:30:43.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:43.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-43 patch sysinv 2020-05-27 16:30:43.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:43.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-43 patch sysinv 2020-05-27 16:30:44.575 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-44 patch sysinv 2020-05-27 16:30:44.575 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:44.621 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-44 patch sysinv 2020-05-27 16:30:44.621 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:48.499 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:30:48.500 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:30:48.500 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:30:48.501 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:30:48.501 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:30:54.613 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-54 patch sysinv 2020-05-27 16:30:54.613 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:54.665 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-54 patch sysinv 2020-05-27 16:30:54.665 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:55.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-55 patch sysinv 2020-05-27 16:30:55.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:55.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-55 patch sysinv 2020-05-27 16:30:55.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-55 patch sysinv 2020-05-27 16:30:55.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:30:55.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-30-55 patch sysinv 2020-05-27 16:30:55.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:30:55.391 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-30-55 patch sysinv 2020-05-27 16:31:01.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-01 patch sysinv 2020-05-27 16:31:01.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:01.733 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-01 patch sysinv 2020-05-27 16:31:01.789 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-01 patch sysinv 2020-05-27 16:31:01.789 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:01.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-01 patch sysinv 2020-05-27 16:31:01.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:01.900 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-01 patch sysinv 2020-05-27 16:31:04.618 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-04 patch sysinv 2020-05-27 16:31:04.618 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:04.662 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-04 patch sysinv 2020-05-27 16:31:04.663 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:11.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-11 patch sysinv 2020-05-27 16:31:11.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:11.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-11 patch sysinv 2020-05-27 16:31:11.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-11 patch sysinv 2020-05-27 16:31:11.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:11.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-11 patch sysinv 2020-05-27 16:31:11.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:11.888 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-11 patch sysinv 2020-05-27 16:31:14.648 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-14 patch sysinv 2020-05-27 16:31:14.649 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:14.705 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-14 patch sysinv 2020-05-27 16:31:14.706 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:17.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-17 patch sysinv 2020-05-27 16:31:17.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:17.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-17 patch sysinv 2020-05-27 16:31:17.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-17 patch sysinv 2020-05-27 16:31:17.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:17.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-17 patch sysinv 2020-05-27 16:31:17.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:17.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-17 patch sysinv 2020-05-27 16:31:24.659 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-24 patch sysinv 2020-05-27 16:31:24.659 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:24.704 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-24 patch sysinv 2020-05-27 16:31:24.704 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:28.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-28 patch sysinv 2020-05-27 16:31:28.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:28.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-28 patch sysinv 2020-05-27 16:31:28.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-28 patch sysinv 2020-05-27 16:31:28.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:28.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-28 patch sysinv 2020-05-27 16:31:28.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:28.887 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-28 patch sysinv 2020-05-27 16:31:34.679 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-34 patch sysinv 2020-05-27 16:31:34.679 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:34.725 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-34 patch sysinv 2020-05-27 16:31:34.725 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:42.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-42 patch sysinv 2020-05-27 16:31:42.721 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:42.744 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-42 patch sysinv 2020-05-27 16:31:42.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-42 patch sysinv 2020-05-27 16:31:42.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:42.832 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-42 patch sysinv 2020-05-27 16:31:42.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:42.899 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-42 patch sysinv 2020-05-27 16:31:44.698 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-44 patch sysinv 2020-05-27 16:31:44.698 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:44.749 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-44 patch sysinv 2020-05-27 16:31:44.749 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:48.568 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:31:48.569 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:31:48.570 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:31:48.570 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:31:48.571 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:31:51.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-51 patch sysinv 2020-05-27 16:31:51.221 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:51.243 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-51 patch sysinv 2020-05-27 16:31:51.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-51 patch sysinv 2020-05-27 16:31:51.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:51.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-51 patch sysinv 2020-05-27 16:31:51.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:51.419 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-51 patch sysinv 2020-05-27 16:31:53.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-53 patch sysinv 2020-05-27 16:31:53.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:53.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-53 patch sysinv 2020-05-27 16:31:53.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-53 patch sysinv 2020-05-27 16:31:53.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:53.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-53 patch sysinv 2020-05-27 16:31:53.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:31:53.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-31-53 patch sysinv 2020-05-27 16:31:54.735 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-54 patch sysinv 2020-05-27 16:31:54.736 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:31:54.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-31-54 patch sysinv 2020-05-27 16:31:54.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:02.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-02 patch sysinv 2020-05-27 16:32:02.721 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:02.743 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-02 patch sysinv 2020-05-27 16:32:02.811 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-02 patch sysinv 2020-05-27 16:32:02.812 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:02.853 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-02 patch sysinv 2020-05-27 16:32:02.853 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:02.910 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-02 patch sysinv 2020-05-27 16:32:04.738 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-04 patch sysinv 2020-05-27 16:32:04.739 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:04.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-04 patch sysinv 2020-05-27 16:32:04.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:14.763 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-14 patch sysinv 2020-05-27 16:32:14.763 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:14.807 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-14 patch sysinv 2020-05-27 16:32:14.807 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:24.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:24.745 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:24.832 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:24.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:24.955 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:25.008 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-24 patch sysinv 2020-05-27 16:32:25.008 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:25.221 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-25 patch sysinv 2020-05-27 16:32:25.221 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:25.240 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-25 patch sysinv 2020-05-27 16:32:25.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-25 patch sysinv 2020-05-27 16:32:25.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:25.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-25 patch sysinv 2020-05-27 16:32:25.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:25.398 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-25 patch sysinv 2020-05-27 16:32:34.795 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-34 patch sysinv 2020-05-27 16:32:34.795 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:34.844 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-34 patch sysinv 2020-05-27 16:32:34.844 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:35.213 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-35 patch sysinv 2020-05-27 16:32:35.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:35.234 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-35 patch sysinv 2020-05-27 16:32:35.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-35 patch sysinv 2020-05-27 16:32:35.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:35.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-35 patch sysinv 2020-05-27 16:32:35.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:35.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-35 patch sysinv 2020-05-27 16:32:37.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-37 patch sysinv 2020-05-27 16:32:37.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:37.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-37 patch sysinv 2020-05-27 16:32:37.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-37 patch sysinv 2020-05-27 16:32:37.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:37.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-37 patch sysinv 2020-05-27 16:32:37.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:37.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-37 patch sysinv 2020-05-27 16:32:44.819 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-44 patch sysinv 2020-05-27 16:32:44.819 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:44.864 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-44 patch sysinv 2020-05-27 16:32:44.864 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:48.648 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:32:48.649 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:32:48.649 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:32:48.650 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:32:48.650 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:32:54.852 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-54 patch sysinv 2020-05-27 16:32:54.852 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:54.886 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-54 patch sysinv 2020-05-27 16:32:54.886 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:58.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-58 patch sysinv 2020-05-27 16:32:58.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:58.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-58 patch sysinv 2020-05-27 16:32:58.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-58 patch sysinv 2020-05-27 16:32:58.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:32:58.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-32-58 patch sysinv 2020-05-27 16:32:58.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:32:58.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-32-58 patch sysinv 2020-05-27 16:33:04.857 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-04 patch sysinv 2020-05-27 16:33:04.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:04.905 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-04 patch sysinv 2020-05-27 16:33:04.905 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:06.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-06 patch sysinv 2020-05-27 16:33:06.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:06.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-06 patch sysinv 2020-05-27 16:33:06.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-06 patch sysinv 2020-05-27 16:33:06.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:06.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-06 patch sysinv 2020-05-27 16:33:06.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:06.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-06 patch sysinv 2020-05-27 16:33:11.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-11 patch sysinv 2020-05-27 16:33:11.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:11.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-11 patch sysinv 2020-05-27 16:33:11.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-11 patch sysinv 2020-05-27 16:33:11.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:11.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-11 patch sysinv 2020-05-27 16:33:11.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:11.997 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-11 patch sysinv 2020-05-27 16:33:14.878 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-14 patch sysinv 2020-05-27 16:33:14.878 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:14.925 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-14 patch sysinv 2020-05-27 16:33:14.925 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:17.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-17 patch sysinv 2020-05-27 16:33:17.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:17.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-17 patch sysinv 2020-05-27 16:33:17.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-17 patch sysinv 2020-05-27 16:33:17.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:17.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-17 patch sysinv 2020-05-27 16:33:17.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:17.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-17 patch sysinv 2020-05-27 16:33:24.899 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-24 patch sysinv 2020-05-27 16:33:24.899 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:24.947 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-24 patch sysinv 2020-05-27 16:33:24.948 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:34.722 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.722 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:34.744 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:34.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:34.898 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.986 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-34 patch sysinv 2020-05-27 16:33:34.987 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:35.031 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-35 patch sysinv 2020-05-27 16:33:35.031 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:44.938 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-44 patch sysinv 2020-05-27 16:33:44.938 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:44.985 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-44 patch sysinv 2020-05-27 16:33:44.986 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:46.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-46 patch sysinv 2020-05-27 16:33:46.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:46.237 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-46 patch sysinv 2020-05-27 16:33:46.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-46 patch sysinv 2020-05-27 16:33:46.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:46.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-46 patch sysinv 2020-05-27 16:33:46.337 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:46.393 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-46 patch sysinv 2020-05-27 16:33:47.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-47 patch sysinv 2020-05-27 16:33:47.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:47.239 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-47 patch sysinv 2020-05-27 16:33:47.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-47 patch sysinv 2020-05-27 16:33:47.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:47.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-47 patch sysinv 2020-05-27 16:33:47.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:47.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-47 patch sysinv 2020-05-27 16:33:48.730 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:33:48.731 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:33:48.732 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:33:48.732 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:33:48.733 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:33:54.975 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-54 patch sysinv 2020-05-27 16:33:54.975 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:55.026 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-55 patch sysinv 2020-05-27 16:33:55.027 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:57.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-57 patch sysinv 2020-05-27 16:33:57.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:57.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-57 patch sysinv 2020-05-27 16:33:57.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-57 patch sysinv 2020-05-27 16:33:57.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:33:57.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-33-57 patch sysinv 2020-05-27 16:33:57.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:33:57.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-33-57 patch sysinv 2020-05-27 16:34:04.982 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-04 patch sysinv 2020-05-27 16:34:04.983 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:05.028 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-05 patch sysinv 2020-05-27 16:34:05.028 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:08.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-08 patch sysinv 2020-05-27 16:34:08.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:08.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-08 patch sysinv 2020-05-27 16:34:08.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-08 patch sysinv 2020-05-27 16:34:08.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:08.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-08 patch sysinv 2020-05-27 16:34:08.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:08.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-08 patch sysinv 2020-05-27 16:34:15.000 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-14 patch sysinv 2020-05-27 16:34:15.001 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:15.049 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-15 patch sysinv 2020-05-27 16:34:15.049 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:19.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-19 patch sysinv 2020-05-27 16:34:19.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:19.737 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-19 patch sysinv 2020-05-27 16:34:19.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-19 patch sysinv 2020-05-27 16:34:19.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:19.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-19 patch sysinv 2020-05-27 16:34:19.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:19.894 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-19 patch sysinv 2020-05-27 16:34:25.024 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-25 patch sysinv 2020-05-27 16:34:25.024 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:25.070 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-25 patch sysinv 2020-05-27 16:34:25.071 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:28.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-28 patch sysinv 2020-05-27 16:34:28.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:28.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-28 patch sysinv 2020-05-27 16:34:28.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-28 patch sysinv 2020-05-27 16:34:28.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:28.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-28 patch sysinv 2020-05-27 16:34:28.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:28.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-28 patch sysinv 2020-05-27 16:34:35.043 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-35 patch sysinv 2020-05-27 16:34:35.043 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:35.091 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-35 patch sysinv 2020-05-27 16:34:35.091 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:39.228 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-39 patch sysinv 2020-05-27 16:34:39.229 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:39.254 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-39 patch sysinv 2020-05-27 16:34:39.302 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-39 patch sysinv 2020-05-27 16:34:39.302 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:39.353 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-39 patch sysinv 2020-05-27 16:34:39.353 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:39.516 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-39 patch sysinv 2020-05-27 16:34:42.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-42 patch sysinv 2020-05-27 16:34:42.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:42.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-42 patch sysinv 2020-05-27 16:34:42.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-42 patch sysinv 2020-05-27 16:34:42.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:42.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-42 patch sysinv 2020-05-27 16:34:42.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:42.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-42 patch sysinv 2020-05-27 16:34:45.063 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-45 patch sysinv 2020-05-27 16:34:45.063 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:45.111 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-45 patch sysinv 2020-05-27 16:34:45.112 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:48.710 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:34:48.711 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:34:48.712 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:34:48.712 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:34:48.713 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:34:53.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-53 patch sysinv 2020-05-27 16:34:53.221 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:53.240 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-53 patch sysinv 2020-05-27 16:34:53.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-53 patch sysinv 2020-05-27 16:34:53.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:53.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-53 patch sysinv 2020-05-27 16:34:53.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:34:53.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-34-53 patch sysinv 2020-05-27 16:34:55.101 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-55 patch sysinv 2020-05-27 16:34:55.101 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:34:55.153 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-34-55 patch sysinv 2020-05-27 16:34:55.153 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:05.108 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-05 patch sysinv 2020-05-27 16:35:05.108 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:05.153 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-05 patch sysinv 2020-05-27 16:35:05.153 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:10.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-10 patch sysinv 2020-05-27 16:35:10.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:10.748 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-10 patch sysinv 2020-05-27 16:35:10.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-10 patch sysinv 2020-05-27 16:35:10.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:10.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-10 patch sysinv 2020-05-27 16:35:10.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:10.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-10 patch sysinv 2020-05-27 16:35:15.126 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.127 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:15.176 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.177 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:15.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:15.741 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.788 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:15.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:15.838 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:15.893 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-15 patch sysinv 2020-05-27 16:35:21.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-21 patch sysinv 2020-05-27 16:35:21.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:21.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-21 patch sysinv 2020-05-27 16:35:21.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-21 patch sysinv 2020-05-27 16:35:21.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:21.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-21 patch sysinv 2020-05-27 16:35:21.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:21.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-21 patch sysinv 2020-05-27 16:35:25.152 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-25 patch sysinv 2020-05-27 16:35:25.153 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:25.202 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-25 patch sysinv 2020-05-27 16:35:25.202 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:27.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-27 patch sysinv 2020-05-27 16:35:27.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:27.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-27 patch sysinv 2020-05-27 16:35:27.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-27 patch sysinv 2020-05-27 16:35:27.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:27.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-27 patch sysinv 2020-05-27 16:35:27.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:27.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-27 patch sysinv 2020-05-27 16:35:35.169 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-35 patch sysinv 2020-05-27 16:35:35.170 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:35.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-35 patch sysinv 2020-05-27 16:35:35.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:45.192 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-45 patch sysinv 2020-05-27 16:35:45.192 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:45.240 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-45 patch sysinv 2020-05-27 16:35:45.241 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:48.793 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:35:48.794 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:35:48.795 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:35:48.795 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:35:48.796 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:35:49.718 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-49 patch sysinv 2020-05-27 16:35:49.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:49.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-49 patch sysinv 2020-05-27 16:35:49.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-49 patch sysinv 2020-05-27 16:35:49.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:49.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-49 patch sysinv 2020-05-27 16:35:49.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:49.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-49 patch sysinv 2020-05-27 16:35:52.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-52 patch sysinv 2020-05-27 16:35:52.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:52.742 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-52 patch sysinv 2020-05-27 16:35:52.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-52 patch sysinv 2020-05-27 16:35:52.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:52.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-52 patch sysinv 2020-05-27 16:35:52.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:35:52.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-35-52 patch sysinv 2020-05-27 16:35:55.230 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-55 patch sysinv 2020-05-27 16:35:55.230 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:35:55.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-35-55 patch sysinv 2020-05-27 16:35:55.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:00.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-00 patch sysinv 2020-05-27 16:36:00.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:00.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-00 patch sysinv 2020-05-27 16:36:00.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-00 patch sysinv 2020-05-27 16:36:00.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:00.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-00 patch sysinv 2020-05-27 16:36:00.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:00.998 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-00 patch sysinv 2020-05-27 16:36:03.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-03 patch sysinv 2020-05-27 16:36:03.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:03.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-03 patch sysinv 2020-05-27 16:36:03.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-03 patch sysinv 2020-05-27 16:36:03.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:03.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-03 patch sysinv 2020-05-27 16:36:03.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:03.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-03 patch sysinv 2020-05-27 16:36:05.236 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-05 patch sysinv 2020-05-27 16:36:05.237 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:05.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-05 patch sysinv 2020-05-27 16:36:05.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:15.256 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-15 patch sysinv 2020-05-27 16:36:15.256 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:15.301 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-15 patch sysinv 2020-05-27 16:36:15.301 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:23.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-23 patch sysinv 2020-05-27 16:36:23.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:23.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-23 patch sysinv 2020-05-27 16:36:23.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-23 patch sysinv 2020-05-27 16:36:23.286 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:23.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-23 patch sysinv 2020-05-27 16:36:23.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:23.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-23 patch sysinv 2020-05-27 16:36:25.274 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-25 patch sysinv 2020-05-27 16:36:25.275 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:25.329 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-25 patch sysinv 2020-05-27 16:36:25.329 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:34.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-34 patch sysinv 2020-05-27 16:36:34.220 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:34.239 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-34 patch sysinv 2020-05-27 16:36:34.276 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-34 patch sysinv 2020-05-27 16:36:34.276 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:34.326 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-34 patch sysinv 2020-05-27 16:36:34.327 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:34.382 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-34 patch sysinv 2020-05-27 16:36:35.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:35.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.281 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.281 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:35.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:35.386 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.457 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.457 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:35.505 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-35 patch sysinv 2020-05-27 16:36:35.506 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:44.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-44 patch sysinv 2020-05-27 16:36:44.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:44.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-44 patch sysinv 2020-05-27 16:36:44.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-44 patch sysinv 2020-05-27 16:36:44.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:44.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-44 patch sysinv 2020-05-27 16:36:44.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:44.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-44 patch sysinv 2020-05-27 16:36:45.317 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-45 patch sysinv 2020-05-27 16:36:45.318 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:45.374 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-45 patch sysinv 2020-05-27 16:36:45.374 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:48.879 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:36:48.880 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:36:48.880 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:36:48.881 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:36:48.881 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:36:55.352 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-55 patch sysinv 2020-05-27 16:36:55.352 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:55.405 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-55 patch sysinv 2020-05-27 16:36:55.406 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:57.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-57 patch sysinv 2020-05-27 16:36:57.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:57.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-57 patch sysinv 2020-05-27 16:36:57.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-57 patch sysinv 2020-05-27 16:36:57.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:36:57.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-36-57 patch sysinv 2020-05-27 16:36:57.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:36:57.892 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-36-57 patch sysinv 2020-05-27 16:37:05.405 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-05 patch sysinv 2020-05-27 16:37:05.405 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:05.452 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-05 patch sysinv 2020-05-27 16:37:05.453 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:09.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-09 patch sysinv 2020-05-27 16:37:09.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:09.734 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-09 patch sysinv 2020-05-27 16:37:09.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-09 patch sysinv 2020-05-27 16:37:09.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:09.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-09 patch sysinv 2020-05-27 16:37:09.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:09.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-09 patch sysinv 2020-05-27 16:37:15.377 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.378 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:15.428 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.429 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:15.713 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:15.731 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:15.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:15.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:15.991 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-15 patch sysinv 2020-05-27 16:37:25.401 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-25 patch sysinv 2020-05-27 16:37:25.401 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:25.456 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-25 patch sysinv 2020-05-27 16:37:25.457 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:26.225 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-26 patch sysinv 2020-05-27 16:37:26.225 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:26.243 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-26 patch sysinv 2020-05-27 16:37:26.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-26 patch sysinv 2020-05-27 16:37:26.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:26.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-26 patch sysinv 2020-05-27 16:37:26.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:26.387 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-26 patch sysinv 2020-05-27 16:37:31.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-31 patch sysinv 2020-05-27 16:37:31.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:31.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-31 patch sysinv 2020-05-27 16:37:31.788 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-31 patch sysinv 2020-05-27 16:37:31.788 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:31.840 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-31 patch sysinv 2020-05-27 16:37:31.840 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:31.905 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-31 patch sysinv 2020-05-27 16:37:35.419 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-35 patch sysinv 2020-05-27 16:37:35.420 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:35.471 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-35 patch sysinv 2020-05-27 16:37:35.472 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:43.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-43 patch sysinv 2020-05-27 16:37:43.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:43.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-43 patch sysinv 2020-05-27 16:37:43.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-43 patch sysinv 2020-05-27 16:37:43.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:43.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-43 patch sysinv 2020-05-27 16:37:43.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:43.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-43 patch sysinv 2020-05-27 16:37:45.445 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-45 patch sysinv 2020-05-27 16:37:45.446 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:45.492 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-45 patch sysinv 2020-05-27 16:37:45.493 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:48.958 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:37:48.959 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:37:48.959 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:37:48.960 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:37:48.960 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:37:55.475 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-55 patch sysinv 2020-05-27 16:37:55.476 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:55.507 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-55 patch sysinv 2020-05-27 16:37:55.508 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:56.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-56 patch sysinv 2020-05-27 16:37:56.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:56.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-56 patch sysinv 2020-05-27 16:37:56.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-56 patch sysinv 2020-05-27 16:37:56.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:37:56.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-37-56 patch sysinv 2020-05-27 16:37:56.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:37:56.891 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-37-56 patch sysinv 2020-05-27 16:38:05.223 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.224 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:05.248 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.294 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.294 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:05.357 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.357 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:05.424 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.493 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.494 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:05.543 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-05 patch sysinv 2020-05-27 16:38:05.544 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:07.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-07 patch sysinv 2020-05-27 16:38:07.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:07.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-07 patch sysinv 2020-05-27 16:38:07.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-07 patch sysinv 2020-05-27 16:38:07.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:07.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-07 patch sysinv 2020-05-27 16:38:07.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:07.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-07 patch sysinv 2020-05-27 16:38:15.501 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-15 patch sysinv 2020-05-27 16:38:15.501 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:15.553 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-15 patch sysinv 2020-05-27 16:38:15.553 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:19.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-19 patch sysinv 2020-05-27 16:38:19.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:19.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-19 patch sysinv 2020-05-27 16:38:19.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-19 patch sysinv 2020-05-27 16:38:19.280 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:19.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-19 patch sysinv 2020-05-27 16:38:19.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:19.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-19 patch sysinv 2020-05-27 16:38:25.523 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-25 patch sysinv 2020-05-27 16:38:25.523 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:25.572 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-25 patch sysinv 2020-05-27 16:38:25.572 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:35.544 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-35 patch sysinv 2020-05-27 16:38:35.545 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:35.591 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-35 patch sysinv 2020-05-27 16:38:35.592 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:38.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-38 patch sysinv 2020-05-27 16:38:38.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:38.739 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-38 patch sysinv 2020-05-27 16:38:38.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-38 patch sysinv 2020-05-27 16:38:38.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:38.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-38 patch sysinv 2020-05-27 16:38:38.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:38.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-38 patch sysinv 2020-05-27 16:38:40.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-40 patch sysinv 2020-05-27 16:38:40.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:40.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-40 patch sysinv 2020-05-27 16:38:40.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-40 patch sysinv 2020-05-27 16:38:40.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:40.833 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-40 patch sysinv 2020-05-27 16:38:40.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:40.889 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-40 patch sysinv 2020-05-27 16:38:45.573 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-45 patch sysinv 2020-05-27 16:38:45.573 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:45.615 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-45 patch sysinv 2020-05-27 16:38:45.615 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:48.934 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:38:48.935 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:38:48.936 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:38:48.937 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:38:48.937 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:38:49.218 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-49 patch sysinv 2020-05-27 16:38:49.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:49.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-49 patch sysinv 2020-05-27 16:38:49.281 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-49 patch sysinv 2020-05-27 16:38:49.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:49.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-49 patch sysinv 2020-05-27 16:38:49.332 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:49.388 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-49 patch sysinv 2020-05-27 16:38:53.719 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-53 patch sysinv 2020-05-27 16:38:53.720 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:53.738 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-53 patch sysinv 2020-05-27 16:38:53.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-53 patch sysinv 2020-05-27 16:38:53.784 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:53.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-53 patch sysinv 2020-05-27 16:38:53.835 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:38:53.998 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-38-53 patch sysinv 2020-05-27 16:38:55.598 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-55 patch sysinv 2020-05-27 16:38:55.598 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:38:55.636 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-38-55 patch sysinv 2020-05-27 16:38:55.636 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:05.636 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-05 patch sysinv 2020-05-27 16:39:05.636 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:05.682 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-05 patch sysinv 2020-05-27 16:39:05.682 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:15.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.214 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:15.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:15.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:15.397 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.631 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.632 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:15.674 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-15 patch sysinv 2020-05-27 16:39:15.674 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:19.715 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-19 patch sysinv 2020-05-27 16:39:19.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:19.735 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-19 patch sysinv 2020-05-27 16:39:19.789 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-19 patch sysinv 2020-05-27 16:39:19.789 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:19.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-19 patch sysinv 2020-05-27 16:39:19.834 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:19.890 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-19 patch sysinv 2020-05-27 16:39:25.652 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-25 patch sysinv 2020-05-27 16:39:25.652 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:25.700 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-25 patch sysinv 2020-05-27 16:39:25.700 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:28.716 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-28 patch sysinv 2020-05-27 16:39:28.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:28.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-28 patch sysinv 2020-05-27 16:39:28.785 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-28 patch sysinv 2020-05-27 16:39:28.786 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:28.839 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-28 patch sysinv 2020-05-27 16:39:28.839 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:28.893 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-28 patch sysinv 2020-05-27 16:39:30.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-30 patch sysinv 2020-05-27 16:39:30.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:30.233 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-30 patch sysinv 2020-05-27 16:39:30.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-30 patch sysinv 2020-05-27 16:39:30.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:30.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-30 patch sysinv 2020-05-27 16:39:30.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:30.393 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-30 patch sysinv 2020-05-27 16:39:35.674 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-35 patch sysinv 2020-05-27 16:39:35.674 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:35.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-35 patch sysinv 2020-05-27 16:39:35.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:45.688 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-45 patch sysinv 2020-05-27 16:39:45.689 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:45.744 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-45 patch sysinv 2020-05-27 16:39:45.744 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:49.217 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:39:49.218 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:39:49.219 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:39:49.219 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:39:49.220 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:39:50.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-50 patch sysinv 2020-05-27 16:39:50.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:50.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-50 patch sysinv 2020-05-27 16:39:50.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-50 patch sysinv 2020-05-27 16:39:50.285 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:50.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-50 patch sysinv 2020-05-27 16:39:50.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:39:50.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-39-50 patch sysinv 2020-05-27 16:39:55.722 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-55 patch sysinv 2020-05-27 16:39:55.723 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:39:55.777 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-39-55 patch sysinv 2020-05-27 16:39:55.777 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:01.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-01 patch sysinv 2020-05-27 16:40:01.219 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:01.238 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-01 patch sysinv 2020-05-27 16:40:01.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-01 patch sysinv 2020-05-27 16:40:01.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:01.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-01 patch sysinv 2020-05-27 16:40:01.336 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:01.400 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-01 patch sysinv 2020-05-27 16:40:02.727 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-02 patch sysinv 2020-05-27 16:40:02.728 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:02.757 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-02 patch sysinv 2020-05-27 16:40:02.806 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-02 patch sysinv 2020-05-27 16:40:02.806 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:02.856 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-02 patch sysinv 2020-05-27 16:40:02.856 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:02.911 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-02 patch sysinv 2020-05-27 16:40:05.740 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-05 patch sysinv 2020-05-27 16:40:05.740 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:05.787 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-05 patch sysinv 2020-05-27 16:40:05.788 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:11.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-11 patch sysinv 2020-05-27 16:40:11.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:11.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-11 patch sysinv 2020-05-27 16:40:11.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-11 patch sysinv 2020-05-27 16:40:11.284 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:11.333 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-11 patch sysinv 2020-05-27 16:40:11.334 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:11.389 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-11 patch sysinv 2020-05-27 16:40:15.750 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-15 patch sysinv 2020-05-27 16:40:15.751 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:15.797 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-15 patch sysinv 2020-05-27 16:40:15.798 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:25.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:25.726 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:25.747 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:25.803 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:25.804 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:25.865 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:25.866 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:25.938 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:26.013 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-25 patch sysinv 2020-05-27 16:40:26.014 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:26.061 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-26 patch sysinv 2020-05-27 16:40:26.061 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:35.799 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-35 patch sysinv 2020-05-27 16:40:35.800 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:35.837 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-35 patch sysinv 2020-05-27 16:40:35.838 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:38.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-38 patch sysinv 2020-05-27 16:40:38.217 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:38.236 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-38 patch sysinv 2020-05-27 16:40:38.286 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-38 patch sysinv 2020-05-27 16:40:38.286 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:38.456 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-38 patch sysinv 2020-05-27 16:40:38.457 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:38.511 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-38 patch sysinv 2020-05-27 16:40:42.721 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-42 patch sysinv 2020-05-27 16:40:42.722 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:42.740 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-42 patch sysinv 2020-05-27 16:40:42.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-42 patch sysinv 2020-05-27 16:40:42.783 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:42.838 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-42 patch sysinv 2020-05-27 16:40:42.838 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:42.898 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-42 patch sysinv 2020-05-27 16:40:45.815 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-45 patch sysinv 2020-05-27 16:40:45.816 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:45.870 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-45 patch sysinv 2020-05-27 16:40:45.871 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:49.099 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:40:49.100 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:40:49.101 90332 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)': /api/v1/nodes: NewConnectionError: : Failed to establish a new connection: [Errno 111] ECONNREFUSED sysinv 2020-05-27 16:40:49.102 90332 ERROR sysinv.common.kubernetes [-] Kubernetes exception in kube_get_nodes: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) sysinv 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._conductor_audit: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)): MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task task(self, context) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4833, in _conductor_audit 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task self._audit_kubernetes_labels(hosts) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 4848, in _audit_kubernetes_labels 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task nodes = self._kube.kube_get_nodes() 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 106, in kube_get_nodes 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task api_response = self._get_kubernetesclient_core().list_node() 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13437, in list_node 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task (data) = self.list_node_with_http_info(**kwargs) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13534, in list_node_with_http_info 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task collection_formats=collection_formats) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task _request_timeout=_request_timeout) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task query_params=query_params) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 205, in request 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task headers=headers) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task **urlopen_kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task return self.urlopen(method, url, **extra_kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task response = conn.urlopen(method, u.request_uri, **kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 678, in urlopen 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task **response_kw) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 649, in urlopen 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task _stacktrace=sys.exc_info()[2]) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task MaxRetryError: HTTPSConnectionPool(host='192.168.206.1', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-05-27 16:40:49.102 90332 ERROR sysinv.openstack.common.periodic_task sysinv 2020-05-27 16:40:52.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-52 patch sysinv 2020-05-27 16:40:52.714 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:52.732 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-52 patch sysinv 2020-05-27 16:40:52.796 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-52 patch sysinv 2020-05-27 16:40:52.797 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:52.855 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-52 patch sysinv 2020-05-27 16:40:52.855 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:40:52.911 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-40-52 patch sysinv 2020-05-27 16:40:55.849 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-55 patch sysinv 2020-05-27 16:40:55.849 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:40:55.902 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-40-55 patch sysinv 2020-05-27 16:40:55.903 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:00.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-00 patch sysinv 2020-05-27 16:41:00.717 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:41:00.736 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-41-00 patch sysinv 2020-05-27 16:41:00.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-00 patch sysinv 2020-05-27 16:41:00.782 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:00.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-00 patch sysinv 2020-05-27 16:41:00.836 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:41:00.894 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-41-00 patch sysinv 2020-05-27 16:41:05.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-05 patch sysinv 2020-05-27 16:41:05.858 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:05.901 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-05 patch sysinv 2020-05-27 16:41:05.902 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:13.215 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-13 patch sysinv 2020-05-27 16:41:13.216 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:41:13.235 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-41-13 patch sysinv 2020-05-27 16:41:13.282 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-13 patch sysinv 2020-05-27 16:41:13.283 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:13.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-13 patch sysinv 2020-05-27 16:41:13.335 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 1. delta_handle ['availability', 'operational'] sysinv 2020-05-27 16:41:13.390 90935 INFO sysinv.api.controllers.v1.host [-] host controller-0 ihost_patch_end_2020-05-27-16-41-13 patch sysinv 2020-05-27 16:41:15.879 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-15 patch sysinv 2020-05-27 16:41:15.880 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0. sysinv 2020-05-27 16:41:15.922 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2020-05-27-16-41-15 patch sysinv 2020-05-27 16:41:15.922 90935 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0.