stein/train standalone upgrade fails when HA is configured

Bug #1901208 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Won't Fix
Wishlist
Unassigned

Bug Description

When running standalone on centos7 between Rocky and Stein or Stein and Train, the upgrade fails because docker is disabled.

2020-10-21 03:07:42 | 2020-10-21 03:07:42.221 317589 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] TASK [Online data migration for Cinder] ****************************************
2020-10-21 03:07:42 | 2020-10-21 03:07:42.223 317589 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [undercloud -> 192.168.24.1]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "cinder_api", "cinder-manage", "db", "online_data_migrations"], "delta": "0:00:00.055891", "end": "2020-10-21 03:07:42.199657", "msg": "non-zero return code", "rc": 1, "start": "2020-10-21 03:07:42.143766", "stderr": "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", "stderr_lines": ["Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"], "stdout": "", "stdout_lines": []}

This is standalone specific because we define the OS::TripleO::Services::Podman service in environments/standalone/standalone-tripleo.yaml but we use container_cli: docker via environments/docker-ha.yaml. We don't then disable the Podman service definition so both podman and docker are configured. During the upgrade, the podman service nukes docker so it breaks the upgrade.

While we could fix this by switch docker-ha.yaml to disable the Podman service, this might break actual deployments depending on environment file loading order (e.g. podman.yaml is defined prior to docker-ha.yaml). Since this only affects upstream standalone CI, we should probably just stop running the upgrade standalone jobs on centos7 and document this via this bug. We don't officially support this for end users and we no longer wish to continue to manage the centos7 upgrade jobs since they don't provide us necessary coverage anymore now that we have centos8 jobs starting in Train. The centos7 -> centos8 (or rhel) upgrade won't hit this because we completely remove packages via leapp so you cannot hit this situation normally.

Changed in tripleo:
status: Triaged → Won't Fix
importance: High → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759476

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/759477

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759478

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/759480

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759482

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/759484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-upgrade (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759485

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-upgrade (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/759486

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759489

summary: - stein upgrade fails when HA is configured
+ stein/train standalone upgrade fails when HA is configured
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (stable/train)

Reviewed: https://review.opendev.org/759489
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=5190211dae5cbd442bd48abd2da58aa639ddeb2b
Submitter: Zuul
Branch: stable/train

commit 5190211dae5cbd442bd48abd2da58aa639ddeb2b
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 11:04:49 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: Ife6231abb92a185af2ccd23358c594dbddf40bbc
    Related-Bug: #1901208

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-upgrade (stable/stein)

Reviewed: https://review.opendev.org/759486
Committed: https://git.openstack.org/cgit/openstack/tripleo-upgrade/commit/?id=2d84f34d4f3db59dc7098be11fde6d8a5bebf47e
Submitter: Zuul
Branch: stable/stein

commit 2d84f34d4f3db59dc7098be11fde6d8a5bebf47e
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:51:17 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: I494bb572167371bda39bb5fdff7f260cdd97d984
    Related-Bug: #1901208

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/759477
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1b626f8a6faab4da4f777a069c3a75a39eb65946
Submitter: Zuul
Branch: stable/stein

commit 1b626f8a6faab4da4f777a069c3a75a39eb65946
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:42:05 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: I05a9f371ca10a4d0012180d763f715363949240a
    Related-Bug: #1901208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/759482
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=4f4524e1059c2bc1af9a193dea2d0072a80f08b5
Submitter: Zuul
Branch: stable/train

commit 4f4524e1059c2bc1af9a193dea2d0072a80f08b5
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:48:41 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: Ibb1400154042106f2cca2f91d5ee3fb7c9eb9091
    Related-Bug: #1901208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/stein)

Reviewed: https://review.opendev.org/759484
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=2ac2b5137eb1177cc5e23c8118d9ad517be7663b
Submitter: Zuul
Branch: stable/stein

commit 2ac2b5137eb1177cc5e23c8118d9ad517be7663b
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:49:23 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: Ibb1400154042106f2cca2f91d5ee3fb7c9eb9091
    Related-Bug: #1901208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-upgrade (stable/train)

Reviewed: https://review.opendev.org/759485
Committed: https://git.openstack.org/cgit/openstack/tripleo-upgrade/commit/?id=58c1ab94b439ca24b545f8d3e8e8840d59e26cd6
Submitter: Zuul
Branch: stable/train

commit 58c1ab94b439ca24b545f8d3e8e8840d59e26cd6
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:50:42 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: I494bb572167371bda39bb5fdff7f260cdd97d984
    Related-Bug: #1901208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/759476
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d6b29f134538c3279c74a5664c48bf8397ad9581
Submitter: Zuul
Branch: stable/train

commit d6b29f134538c3279c74a5664c48bf8397ad9581
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:40:06 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: I05a9f371ca10a4d0012180d763f715363949240a
    Related-Bug: #1901208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/train)

Reviewed: https://review.opendev.org/759478
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=f962b72847e53fec1857e2d0dbd8fe7e76f06380
Submitter: Zuul
Branch: stable/train

commit f962b72847e53fec1857e2d0dbd8fe7e76f06380
Author: Alex Schultz <email address hidden>
Date: Fri Oct 23 10:45:12 2020 -0600

    Remove centos-7-standalone-upgrade

    It no longer provides quality coverage, see bug for details.

    Change-Id: I843aa5111ccbb63e7007f1eb6db93fdfbdb76a88
    Related-Bug: #1901208

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.