Cannot run scenario4 on rhel8 due to missing pcs required by manila

Bug #1841941 reported by Sorin Sbarnea
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Tom Barron

Bug Description

While trying to activate scenario4 testing on rhel8 we discovered that manila requires pcs which is not available.

This is blocking https://tree.taiga.io/project/tripleo-ci-board/task/1264

Test change where failure can be seen: https://review.rdoproject.org/r/#/c/21860/

Sorin Sbarnea (ssbarnea)
description: updated
Revision history for this message
Marios Andreou (marios-b) wrote :

originally discussed there https://review.opendev.org/#/c/636563/ and revived yesterday with tbarron on irc.

for now we plan to remove scenario4 from the zuul layout so it no longer runs until the non pcs scenario is supported

rdo-jobs master: Remove container_cli: docker from scenario 1-3 standalone https://review.rdoproject.org/r/21978

config master: Remove scenario004 standalone rhel-8 job no manila without pcs https://review.rdoproject.org/r/21999

wes hayutin (weshayutin)
Changed in tripleo:
status: Confirmed → Triaged
importance: Undecided → Critical
milestone: none → train-rc1
Revision history for this message
Alan Pevec (apevec) wrote :

> originally discussed there https://review.opendev.org/#/c/636563/

tl;dr
centos7 + pacemaker + podman don't work together, remove Pacemaker from these jobs until centos8 is out, so we have centos8 + podman + pacemaker tested.
http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002730.html

tags: removed: promotion-blocker
Revision history for this message
Marios Andreou (marios-b) wrote :

re-added the periodic as discussed during tripleo-ci scrum today
config master: Re-add periodic-tripleo-ci-rhel-8-scenario004-standalone-master https://review.rdoproject.org/r/22103

tags: added: promotion-blocker
Tom Barron (tpb)
Changed in tripleo:
assignee: nobody → Tom Barron (tpb)
status: Triaged → In Progress
Revision history for this message
Sorin Sbarnea (ssbarnea) wrote :

Glad to hear that Tom, ping me as soon I need to enable scenario004 back for promotions.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/682853

Changed in tripleo:
assignee: Tom Barron (tpb) → Sorin Sbarnea (ssbarnea)
Changed in tripleo:
assignee: Sorin Sbarnea (ssbarnea) → Tom Barron (tpb)
Changed in tripleo:
assignee: Tom Barron (tpb) → wes hayutin (weshayutin)
Changed in tripleo:
assignee: wes hayutin (weshayutin) → Tom Barron (tpb)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/685097

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/685449

Revision history for this message
wes hayutin (weshayutin) wrote :

 thank you both for your patience. I think we have some valid results finally. I had to add the podman template and get it in the right order to override scenario004 https://review.rdoproject.org/r/#/c/22889/1/zuul.d/standalone-jobs.yaml

http://logs.rdoproject.org/53/682853/14/openstack-check/tripleo-ci-rhel-8-scenario004-standalone-rdo/4891f74/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

019-10-03 23:30:16 | "failed: [standalone -> 192.168.24.1] (item=cephfs_data) => changed=false ",
2019-10-03 23:30:16 | " - cephfs_data",
2019-10-03 23:30:16 | " - cephfs",
2019-10-03 23:30:16 | " delta: '0:00:00.529869'",
2019-10-03 23:30:16 | " end: '2019-10-03 23:30:14.994543'",
2019-10-03 23:30:16 | " item: cephfs_data",
2019-10-03 23:30:16 | " start: '2019-10-03 23:30:14.464674'",
2019-10-03 23:30:16 | " Error ENOENT: unrecognized pool 'cephfs_data'",
2019-10-03 23:30:16 | "failed: [standalone -> 192.168.24.1] (item=cephfs_metadata) => changed=false ",
2019-10-03 23:30:16 | " - cephfs_metadata",
2019-10-03 23:30:16 | " delta: '0:00:00.496616'",
2019-10-03 23:30:16 | " end: '2019-10-03 23:30:15.711899'",
2019-10-03 23:30:16 | " item: cephfs_metadata",
2019-10-03 23:30:16 | " start: '2019-10-03 23:30:15.215283'",
2019-10-03 23:30:16 | " Error ENOENT: unrecognized pool 'cephfs_metadata'",

Changed in tripleo:
milestone: train-rc1 → ussuri-1
wes hayutin (weshayutin)
tags: removed: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/685449
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=d3d57757ae02a709aa2eac935d964b9d602e9128
Submitter: Zuul
Branch: master

commit d3d57757ae02a709aa2eac935d964b9d602e9128
Author: John Fulton <email address hidden>
Date: Fri Sep 27 14:06:46 2019 -0400

    Use custom ceph-ansible until c8 storage sig ready

    TripleO scenarios use Ceph packages from CentOS Storage Sig.
    CentOS8 and RHEL8 based TripleO jobs cannot yet use CentOS8
    Storage Sig packages so use a temporary build from rhel8-master
    as a workaround until they are ready.

    Change-Id: Id13ff8bf1d896a039dc29e0db7d36d48c3186bff
    Related-bug: #1841941

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/682853
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cff553c7e651aef769ef0ef7f0fc5e9757467cce
Submitter: Zuul
Branch: master

commit cff553c7e651aef769ef0ef7f0fc5e9757467cce
Author: Tom Barron <email address hidden>
Date: Thu Aug 29 20:38:15 2019 -0400

    Fix Manila when run without pacemaker

    The supported path is to run the manila-share service
    under control of pacemaker using container bundles started
    in Step5. However, it is useful for test and POC purposes
    to be able to run the manila-share service without pacemaker
    control as a regular container started in Step4.

    This change fixes the non-pacemaker deployment path, which runs
    manila-share in Step4 rather than Step5, so that it uses the same
    privileges for the manila-share container as are used for the
    pacemaker-controlled container-bundle in Step5.

    However. we do not change scenario004 to deploy manila using the
    non-pacemaker path since in CI we want to test the path that is
    supported for customers.

    Change-Id: I8effea96a4428281b5a5dfa9711bb602fdbdc705
    Related-Bug: #1841941
    Depends-On: https://review.opendev.org/#/c/685449

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/685097
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=68a4e125bbd02e7ae5f695045c64201e4a7b5c83
Submitter: Zuul
Branch: stable/stein

commit 68a4e125bbd02e7ae5f695045c64201e4a7b5c83
Author: Tom Barron <email address hidden>
Date: Thu Aug 29 20:38:15 2019 -0400

    Fix Manila when run without pacemaker

    The supported path is to run the manila-share service
    under control of pacemaker using container bundles started
    in Step5. However, it is useful for test and POC purposes
    to be able to run the manila-share service without pacemaker
    control as a regular container started in Step4.

    This change fixes the non-pacemaker deployment path, which runs
    manila-share in Step4 rather than Step5, so that it uses the same
    privileges for the manila-share container as are used for the
    pacemaker-controlled container-bundle in Step5.

    However. we do not change scenario004 to deploy manila using the
    non-pacemaker path since in CI we want to test the path that is
    supported for customers.

    Change-Id: I8effea96a4428281b5a5dfa9711bb602fdbdc705
    Related-Bug: #1841941
    (cherry picked from commit cff553c7e651aef769ef0ef7f0fc5e9757467cce)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687674

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/687674
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=24d1d91d8c426f81b5a1ea1e63781b58ae6d9304
Submitter: Zuul
Branch: stable/rocky

commit 24d1d91d8c426f81b5a1ea1e63781b58ae6d9304
Author: Tom Barron <email address hidden>
Date: Thu Aug 29 20:38:15 2019 -0400

    Fix Manila when run without pacemaker

    The supported path is to run the manila-share service
    under control of pacemaker using container bundles started
    in Step5. However, it is useful for test and POC purposes
    to be able to run the manila-share service without pacemaker
    control as a regular container started in Step4.

    This change fixes the non-pacemaker deployment path, which runs
    manila-share in Step4 rather than Step5, so that it uses the same
    privileges for the manila-share container as are used for the
    pacemaker-controlled container-bundle in Step5.

    However. we do not change scenario004 to deploy manila using the
    non-pacemaker path since in CI we want to test the path that is
    supported for customers.

    Change-Id: I8effea96a4428281b5a5dfa9711bb602fdbdc705
    Related-Bug: #1841941
    (cherry picked from commit cff553c7e651aef769ef0ef7f0fc5e9757467cce)
    (cherry picked from commit 68a4e125bbd02e7ae5f695045c64201e4a7b5c83)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/687949

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/687949
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ab76fc8d2dfdc603a47c82e139fee621f4dd6b11
Submitter: Zuul
Branch: stable/queens

commit ab76fc8d2dfdc603a47c82e139fee621f4dd6b11
Author: Tom Barron <email address hidden>
Date: Thu Aug 29 20:38:15 2019 -0400

    Fix Manila when run without pacemaker

    The supported path is to run the manila-share service
    under control of pacemaker using container bundles started
    in Step5. However, it is useful for test and POC purposes
    to be able to run the manila-share service without pacemaker
    control as a regular container started in Step4.

    This change fixes the non-pacemaker deployment path, which runs
    manila-share in Step4 rather than Step5, so that it uses the same
    privileges for the manila-share container as are used for the
    pacemaker-controlled container-bundle in Step5.

    However. we do not change scenario004 to deploy manila using the
    non-pacemaker path since in CI we want to test the path that is
    supported for customers.

    Change-Id: I8effea96a4428281b5a5dfa9711bb602fdbdc705
    Related-Bug: #1841941
    (cherry picked from commit cff553c7e651aef769ef0ef7f0fc5e9757467cce)
    (cherry picked from commit 68a4e125bbd02e7ae5f695045c64201e4a7b5c83)
    (cherry picked from commit 24d1d91d8c426f81b5a1ea1e63781b58ae6d9304)

tags: added: in-stable-queens
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
Revision history for this message
Tom Barron (tpb) wrote :

@sorin, is there an issue here still after the various fixes above merged? I'm thinking that maybe this one can be closed.

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
Revision history for this message
Tom Barron (tpb) wrote :

I believe this is fixed. Sorin, please re-open if there's still somehow an issue here.

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.