Undercloud install failing with ironic-inspector dbsync error

[John Trowbridge]

We have started seeing the undercloud install fail when using quickstart image based deploys (multinode jobs seem unaffected). The issue seems to be some misconfiguration of ironic-inspector, but I have not been able to find any suspicious commits to ironic/inspector/puppet-ironic/instack-undercloud.

Undercloud logs showing the issue: http://chunk.io/f/b98db66e4e8f433b88a90cf04a6b126c

I am creating a local reproducer with quickstart via the command:

tripleo-quickstart/quickstart.sh --clean --tags all --release master-tripleo-ci --teardown all 127.0.0.2

[wes hayutin (weshayutin)]

ironic-inspector trace

http://paste.openstack.org/show/605089/

[John Trowbridge (trown)]

Should be fixed by https://github.com/openstack/ironic-inspector/commit/a509edd993852b56fafce370840f477c36115c48 according to jroll on #openstack-ironic

I think there must be some quickstart bug here though as it does not seem to affect the non-quickstart jobs. http://tripleo.org/cistatus.html

[John Trowbridge (trown)]

It looks like we have an incorrect repo configuration in the tripleo-ci/master release config in quickstart:

https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L33

The commit we need is in the current-tripleo repo, so once https://review.openstack.org/452174 merges we should be good. (and it is in the merge queue now)

I am testing it locally to be sure, since I think wes said it did not help the issue.

[John Trowbridge (trown)]

I tried this locally, and unfortunately https://review.openstack.org/452174 does not help. The reason this is only happening on libvirt is actually because we still have custom code for devmode repo setup in quickstart instead of using the repo-setup role:

https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/undercloud/tasks/inject_repos.yml

We need to change that to use the repo-setup role, but maybe to address the bug we can just fix that custom code.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/452286

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/452286
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=7a7cb6460cc79bf11952fc9304723c20e10fa549
Submitter: Jenkins
Branch: master

commit 7a7cb6460cc79bf11952fc9304723c20e10fa549
Author: John Trowbridge <email address hidden>
Date: Fri Mar 31 18:41:12 2017 +0000

    Switch tripleo-ci to use trunk.rdoproject.org repos

    This is a similar patch to https://review.openstack.org/452174
    We still have this custom code for libvirt devmode, so we need to
    change the repo here too. This is a good reminder that we need to
    remove this custom code and replace it with a call to the repo-setup
    role. This patch addresses the major breakage, but I will post a
    follow-up to refactor this.

    Change-Id: I4d772756a652448203a74f3ca75cacf4ec0d42b6
    Closes-Bug: 1678218

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.