master overcloud deployments are timing out - Exception: Failed to install a bootloader when deploying node

[Ronelle Landy]

Overcloud deployments are timing out on the master promotion - starting on 06/06.

Looking at the overcloud deploy log:

2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_IN_PROGRESS state changed
2019-06-06 02:31:46 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap.NetIpMapValue]: CREATE_COMPLETE state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:44Z [overcloud.CephStorageIpListMap]: CREATE_COMPLETE Stack CREATE completed successfully
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_IN_PROGRESS state changed
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.ObjectStorageAllNodesValidationConfig]: CREATE_IN_PROGRESS Stack CREATE started
2019-06-06 03:49:17 | 2019-06-06 02:31:45Z [overcloud.BlockStorageIpListMap.EnabledServicesValue]: CREATE_COMPLETE state changed

Note the time taken on overcloud.CephStorageIpListMap.

The deployment then times out. The full log is included at:

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bb219fa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The failure is here:

https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/bb219fa/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz#_2019-06-06_02_35_50_233

2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor [req-e69bf577-2f65-4669-a413-8ec7c2b213a5 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''} for node 971480a7-a89d-4d0d-85ce-fba56547a3d7: InstanceDeployFailure: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 362, in heartbeat
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor self.continue_deploy(task)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor return f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 360, in continue_deploy
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor prep_boot_part_uuid=prep_boot_part_uuid)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 743, in prepare_instance_to_boot
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor prep_boot_part_uuid=prep_boot_part_uuid)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 793, in configure_local_boot
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor log_and_raise_deployment_error(task, msg)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 223, in log_and_raise_deployment_error
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor raise exception.InstanceDeployFailure(msg)
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Failed to install a bootloader when deploying node 971480a7-a89d-4d0d-85ce-fba56547a3d7. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
2019-06-06 02:35:50.233 8 ERROR ironic.drivers.modules.agent_base_vendor
2019-06-06 02:35:50.241 8 DEBUG ironic.conductor.task_manager [req-e69bf577-2f65-4

[Ronelle Landy (rlandy)]

Possibly related to https://review.opendev.org/#/c/663361/?

[Ronelle Landy (rlandy)]

<ykarel> started happening after https://review.opendev.org/#/c/639390/

[Dmitry Tantsur (divius)]

Could you check if https://review.opendev.org/#/c/663361/ has fixed this issue?

[Ronelle Landy (rlandy)]

Still seeing this failure in master promotions:

2019-06-07 02:28:08.536 9 ERROR ironic.drivers.modules.agent_base_vendor [req-2e80ee1f-ae6e-4e01-8be4-087e85e28a8e - - - - -] Failed to install a bootloader when deploying node d31a2f95-0649-4681-a381-37ff0e5d61f0. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}

https://review.opendev.org/#/c/663361/ merged the day before:
Jun 6 10:31 PM

[Bob Fournier (bfournie)]

This may be needed - https://review.opendev.org/#/c/663916.

[Ronelle Landy (rlandy)]

Master has promoted since reporting this bug and fs001 has shown passes on:

| 2019-06-10 15:40 | | 7900.0 min | | SUCCESS | | Logs | | openstack-periodic-master |
| 2019-06-10 03:29 | | 7553.0 min | | SUCCESS | | Logs | | openstack-periodic-master |
| 2019-06-09 15:36 | | 7960.0 min | | SUCCESS | | Logs | | openstack-periodic-master |
| 2019-06-09 03:27 | | 7543.0 min | | SUCCESS | | Logs | | openstack-periodic-master |
| 2019-06-08 21:35 | | 8314.0 min | | SUCCESS | | Logs | | openstack-periodic-master |
| 2019-06-08 15:39 | | 7884.0 min | | SUCCESS | | Logs |

so closing this out.