| 2019-06-06 15:04:55 |
Ronelle Landy |
description |
Overcloud deployments are timing out on the master promotion - starting on 06/06.
Looking at the overcloud deploy log:
2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_IN_PROGRESS state changed
2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_COMPLETE state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap]: CREATE_COMPLETE Stack CREATE completed successfully
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_IN_PROGRESS state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.ObjectStorageAllNodesValidationConfig]: CREATE_IN_PROGRESS Stack CREATE started
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_COMPLETE state changed
Note the time taken on overcloud.CephStorageIpListMap.
The deployment then times out. The full log is included at:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bb219fa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz |
Overcloud deployments are timing out on the master promotion - starting on 06/06.
Looking at the overcloud deploy log:
2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_IN_PROGRESS state changed
2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_COMPLETE state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap]: CREATE_COMPLETE Stack CREATE completed successfully
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_IN_PROGRESS state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.ObjectStorageAllNodesValidationConfig]: CREATE_IN_PROGRESS Stack CREATE started
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_COMPLETE state changed
Note the time taken on overcloud.CephStorageIpListMap.
The deployment then times out. The full log is included at:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bb219fa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
The failure is here:
https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bb219fa/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz#_2019-06-06_02_35_50_233
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor [req-e69bf577-2f65-4669-a413-8ec7c2b213a5 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''} for node 971480a7-a89d-4d0d-85ce-fba56547a3d7: InstanceDeployFailure: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 362, in heartbeat
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor self.continue_deploy(task)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor return f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 360, in continue_deploy
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor prep_boot_part_uuid=prep_boot_part_uuid)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 743, in prepare_instance_to_boot
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor prep_boot_part_uuid=prep_boot_part_uuid)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 793, in configure_local_boot
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor log_and_raise_deployment_error(task, msg)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 223, in log_and_raise_deployment_error
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor raise exception.InstanceDeployFailure(msg)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor
2019-06-06 02:35:50.241 8 DEBUG ironic.conductor.task_manager [req-e69bf577-2f65-4 |
|
