periodic- featureset019-queens failing during overcloud deploy with error in tasks: ceph_base_ansible_workflow

[Marios Andreou]

last 4 runs of this periodic promotion job at [1] fail during overcloud deploy with trace like:

        * 2019-12-04 15:01:31 | 2019-12-04 15:01:28Z [overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution]: CREATE_FAILED resources.WorkflowTasks_Step2_Execution: Failure caused by error in tasks: ceph_base_ansible_workflow

        * 2019-12-04 15:01:31 | 2019-12-04 15:01:28Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: Failure caused by error in tasks: ceph_base_ansible_workflow

        * 2019-12-04 15:01:31 | overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
2019-12-04 15:01:31 | resource_type: OS::TripleO::WorkflowSteps
2019-12-04 15:01:31 | physical_resource_id: ec6208d5-7e8a-432a-9677-feb9bfcd9157
2019-12-04 15:01:31 | status: CREATE_FAILED
2019-12-04 15:01:31 | status_reason: |
2019-12-04 15:01:31 | resources.WorkflowTasks_Step2_Execution: Failure caused by error in tasks: ceph_base_ansible_workflow
2019-12-04 15:01:31 |
2019-12-04 15:01:31 | ceph_base_ansible_workflow [task_ex_id=2bb6041c-fa08-470a-a1f1-9373f5a6d3f1] -> Failure caused by error in tasks: ceph_install

Examples there [2][3][4][5]

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens
[2] http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/3ba84bd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3] http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/11aec9f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[4] http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/8032d60/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[5] http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/c7009ad/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Marios Andreou (marios-b)]

revisiting this. I think the error in the description is a red herring it comes from the validations but i don't think it is fatal. The actual error is ceph-related I will update title/description gathering more info now

[Marios Andreou (marios-b)]

My testproject posted yesterday for sanity checking failed in the same way https://review.rdoproject.org/r/23932

 http://logs.rdoproject.org/32/23932/1/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/ba4cec4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Giulio Fidente (gfidente)]

yeah ceph-ansible failing [1], investigating

1. http://logs.rdoproject.org/32/23932/1/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/ba4cec4/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz

[Giulio Fidente (gfidente)]

we need newer container image (3.2.8) to make newer ceph-ansible (3.2.30) to pass

container image version was bumped up by https://review.opendev.org/#/c/692113/ but the job still using 3.2.1 while getting latest ceph-ansible promoted in cbs because it doesn't get pinned with the other osp packages

[Marios Andreou (marios-b)]

thanks for checking gfidente indeed i see ceph_tag: "v3.2.1-stable-3.2-luminous-centos-7-x86_64" in http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/3ba84bd/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz

trying to track where/how/why thanks for your help

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/697442

[Marios Andreou (marios-b)]

still waiting for https://review.opendev.org/697442 stuck in the gate but we managed to get a green run with the test review at https://review.rdoproject.org/r/#/c/23932/ (which included the fix @ 697442) and Queens promoted with that.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697442
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=508eccca5299090070682c991397d827103702d1
Submitter: Zuul
Branch: master

commit 508eccca5299090070682c991397d827103702d1
Author: Marios Andreou <email address hidden>
Date: Thu Dec 5 13:11:26 2019 +0200

    Bump rocky/queens ceph to v3.2.8-stable-3.2-luminous-centos-7

    In [1] this was bumped in tripleo-common but we have this override
    in defaults so we need to change that too.

    Closes-Bug: 1855120
    [1] https://review.opendev.org/692113

    Change-Id: Ib4125e2bd4493e308f405344c32416d608ac5083