Tripleo doesn't handle openshift updates and upgrade correctly

Bug #1810812 reported by Martin André
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Martin André

Bug Description

TripleO unconditionally calls the openshift-ansible's upgrade playbook on upgrade. It may work fine in the most simplistic case (although likely untested), however this is going to break as soon as more complex deployment are in play, such as with CNS.

In the gluster case case, we have upgrade the control plane and gluster nodes in separate steps and run the playbook on each node, one at a time, after running some manual commands [3]. Perhaps it's something we can automate in TripleO, but more realistically fall under openshift-ansible jurisdiction and as of today, they don't offer an automated solution for gluster nodes.

We should remove the upgrade code from TripleO and give a way to the operator follow the steps outlined in the openshift-ansible's documentation. In other words, we need to provide a way to easily call the openshift-ansible playbooks using the TripleO-generated inventory.

[1] https://github.com/openstack/tripleo-heat-templates/blob/fdba8a8/extraconfig/services/openshift-master.yaml#L63
[2] https://github.com/openstack/tripleo-heat-templates/blob/fdba8a8/extraconfig/services/openshift-master.yaml#L362-L364
[3] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#special-considerations-for-glusterfs

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/629016

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/629016
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=97cf1c274a9849e0b1dc84500ca8dae6414a1b83
Submitter: Zuul
Branch: master

commit 97cf1c274a9849e0b1dc84500ca8dae6414a1b83
Author: Martin André <email address hidden>
Date: Mon Jan 7 17:53:30 2019 +0100

    Let the operator manage openshift updates and upgrades

    This remove the upgrade code to ensure that TripleO will not get
    involved in the business of updating and upgrading OpenShift clusters
    as it currently involved manual steps for clusters using glusterfs.

    Instead, the operator is invited to upgrade OpenShift separately from
    TripleO and follow the OpenShift upgrade documentation.

    [1] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#special-considerations-for-glusterfs
    [2] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html

    Change-Id: I6033525cde40fd44d648cef792efbca6901e88f0
    Closes-Bug: #1810812

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/629534

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.3.0

This issue was fixed in the openstack/tripleo-heat-templates 10.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/629534
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3a701cce4a70ea344e9b9268ba157ef81fee90a3
Submitter: Zuul
Branch: stable/rocky

commit 3a701cce4a70ea344e9b9268ba157ef81fee90a3
Author: Martin André <email address hidden>
Date: Mon Jan 7 17:53:30 2019 +0100

    Let the operator manage openshift updates and upgrades

    This remove the upgrade code to ensure that TripleO will not get
    involved in the business of updating and upgrading OpenShift clusters
    as it currently involved manual steps for clusters using glusterfs.

    Instead, the operator is invited to upgrade OpenShift separately from
    TripleO and follow the OpenShift upgrade documentation.

    [1] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#special-considerations-for-glusterfs
    [2] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html

    Change-Id: I6033525cde40fd44d648cef792efbca6901e88f0
    Closes-Bug: #1810812
    (cherry picked from commit 97cf1c274a9849e0b1dc84500ca8dae6414a1b83)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/628960
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d4b78bf5077dd1bbd483919f7796359fa5712fbb
Submitter: Zuul
Branch: master

commit d4b78bf5077dd1bbd483919f7796359fa5712fbb
Author: Martin André <email address hidden>
Date: Mon Jan 7 12:20:29 2019 +0100

    Add ability to run osa playbooks from tripleo-deploy-openshift

    Introduce a `--playbook` option to the `tripleo-deploy-openshift`
    script in order to be able to run openshift-ansible playbook directly
    on already deployed servers.

    It may be necessary to run specific openshift-ansible's playbook for
    certain day 2 operation tasks.

    Related-Bug: #1810812
    Change-Id: Ieb25fa5ac0e209f738ed734e9063175ae2a8aa55

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.3.0

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/643288

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/rocky)

Reviewed: https://review.openstack.org/643288
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=201e9793e17f0ec4eedbe2fac9b82bda6110f6c2
Submitter: Zuul
Branch: stable/rocky

commit 201e9793e17f0ec4eedbe2fac9b82bda6110f6c2
Author: Martin André <email address hidden>
Date: Mon Jan 7 12:20:29 2019 +0100

    Add ability to run osa playbooks from tripleo-deploy-openshift

    Introduce a `--playbook` option to the `tripleo-deploy-openshift`
    script in order to be able to run openshift-ansible playbook directly
    on already deployed servers.

    It may be necessary to run specific openshift-ansible's playbook for
    certain day 2 operation tasks.

    Related-Bug: #1810812
    Change-Id: Ieb25fa5ac0e209f738ed734e9063175ae2a8aa55
    (cherry picked from commit d4b78bf5077dd1bbd483919f7796359fa5712fbb)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.