Overcloud installation fails with "Failure prepping block device." error

[Sagi (Sergey) Shnaidman]

All promotion jobs fail with error from nova: Failure prepping block device
But logs contain errors from a few services - ironic, neutron, nova, heat.

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/1449e0d/undercloud/var/log/nova/nova-compute.log.txt.gz#_2018-02-15_07_15_02_234

nova.virt.ironic.driver [req-b8a80c3d-2b53-4d70-88b8-f1fca88e0a8e 6c793b82f1f443f8854a0a73922e2f7e 4cae382f8b0149c59bc47bcbb0ffa539 - default default] Error deploying instance 41f3a54d-071d-4f13-915f-9e7e427bfd78 on baremetal node f20255f8-7c86-415e-abaf-12350adcba8d.: InstanceDeployFailure: Failed to provision instance 41f3a54d-071d-4f13-915f-9e7e427bfd78: Failed to prepare to deploy: Unable to set binding:host_id for neutron port 72cd4c39-31ad-4427-a208-202903a7e001. Error: Host f20255f8-7c86-415e-abaf-12350adcba8d is not connected to any segments on routed provider network '1e7df6ff-6a06-4a64-9fb2-33cd9f422eef'. It should be connected to one.

2018-02-15 07:15:02.234 3543 ERROR nova.compute.manager [instance: fe98461f-cc41-4ac2-8d9a-3cedd0ef3820] BuildAbortException: Build of instance fe98461f-cc41-4ac2-8d9a-3cedd0ef3820 aborted: Failure prepping block device.

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/1449e0d/undercloud/var/log/neutron/openvswitch-agent.log.txt.gz#_2018-02-15_06_22_15_510

2018-02-15 06:22:15.510 25758 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None
2018-02-15 06:22:15.511 25758 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json] dies due to the error: None

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/1449e0d/undercloud/var/log/ironic/app.log.txt.gz#_2018-02-15_07_14_16_335

2018-02-15 07:14:16.335 ERROR /var/log/ironic/app.log: 26657 ERROR wsme.api [req-c108f25e-19c4-4ff1-b1c8-f32d46e68351 36664fdd6a954d3c8182e9aa7d820d59 4cae382f8b0149c59bc47bcbb0ffa539 - default default] Server-side error: "Unable to attach VIF b4062d77-eed4-4ae1-b08f-dac0bc71723b because Ironic can not update Neutron port b4062d77-eed4-4ae1-b08f-dac0bc71723b MAC address to match physical MAC address fa:16:3e:37:17:f9

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/1449e0d/undercloud/var/log/ironic/ironic-conductor.log.txt.gz#_2018-02-15_07_12_27_830

"2018-02-15T07:12:27.000000Z"}} get_auth_ref /usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py:180
2018-02-15 07:12:27.830 29640 ERROR ironic.drivers.modules.network.flat [req-f41ffa45-7491-4b90-a2d9-fbc2d9bcc495 36664fdd6a954d3c8182e9aa7d820d59 4cae382f8b0149c59bc47bcbb0ffa539 - default default] Unable to set binding:host_id for neutron port 72cd4c39-31ad-4427-a208-202903a7e001. Error: Host f20255f8-7c86-415e-abaf-12350adcba8d is not connected to any segments on routed provider network '1e7df6ff-6a06-4a64-9fb2-33cd9f422eef'. It should be connected to one.

[wes hayutin (weshayutin)]

still happening
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/3e185af/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-02-15_16_01_35

[Harald Jensås (harald-jensas)]

This is happening because we do not have segmentsubnetmapping populated in neutron. To get that we need the ironic-neutron-agent. This change: https://review.openstack.org/#/c/537830

But 537830 change needs a package promote to land.
Lets try to workaround this by making the segment-to-subnet mapping optional.
 https://review.openstack.org/#/c/542215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542215
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=52dbe134abcc20a17e9ee60e4a268df4fa073c5d
Submitter: Zuul
Branch: master

commit 52dbe134abcc20a17e9ee60e4a268df4fa073c5d
Author: Harald Jensas <email address hidden>
Date: Thu Feb 8 14:47:38 2018 +0100

    Make routed networks optional

    Closes-Bug: #1749671
    Change-Id: Ieabb2ded33ec0c0b68f21c9afa16a9baddc61393

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.3.0 release.