[ussuri, victoria, wallaby] openstack-upgrade action fails

[Mihai Gheorghe]

Using pause-single-unit upgrade procedure for Magnum charm, when issuing the openstack-upgrade action after unit pause, the action will fail with:

run_upgrade got an unexpected keyword argument 'upgrade_openstack'

and no upgrade will be issued.
The only way of triggering the update that i found was to edit the openstack_upgrade_action function under
/var/lib/juju/agents/unit-magnum-x/charm/actions/openstack-upgrade

-- charm_instance.run_upgrade(upgrade_openstack=True)
++ charm_instance.run_upgrade()

This will let the upgrade procedure continue.
This also affects Barbican and Placement charms too, so far.

Maybe i am doing something wrong and an extra step is needed besides the upgrade procedure docs.

Magnum charm channel: victoria/edge
Juju version: 2.9.34

[Alex Kavanagh (ajkavanagh)]

> Maybe i am doing something wrong and an extra step is needed besides the upgrade procedure docs.

No, you're not doing anything wrong; this sounds like a bug, where the wrong version of a layer has been brought into the charm.

Please note, that victoria/edge is unstable, and the victoria track has not yet been released into production; that's hopefully coming soon.

[Mihai Gheorghe (m-capsali)]

Well, i am in the process of upgrading Openstack from Ussuri release to Yoga. I just started testing on the lab env. As far as i understand from the docs, the move to charmhub from charmstore requires you to update the charm to the next release of Openstack update. When using charmstore, we could just update the charms to the latest version and than upgrade as many Openstack releases as necesarry. Now, from my understanding, if i must go from Ussuri to Victoria, i must first update the charms to victoria/edge, do Opensatck upgrade and than move to the next charm releas followed by the Openstack upgrade.
Did i misunderstood.
And sorry for polluting the bug report!

[Alex Kavanagh (ajkavanagh)]

Ok, firstly, thank you for testing the charms; it's something our automated tests haven't picked up, so that's really useful information, and something I'm going to look at to resolve.

It's confusing at the moment, because the migration from charmstore (jaas.ai) to charmhub (charmhub.io) is not quite complete (but is almost there). This means that not all the stable tracks in charmhub.io have been released. To recap:

* The charms in latest/stable are the 21.10 charms that support bionic-queens -> focal-xena
* The yoga/stable charms support focal-yoga, jammy-yoga, and an upgrade from focal-xena
* The xena/stable charms support focal-xena, and an upgrade from focal-wallaby.
* The wallaby/stable, victoria/stable and ussuri/stable track/risk combinations have not been released as they haven't quite finished their testing.

So if a charm doesn't have a stable risk against a track (e.g. victoria/stable) then it hasn't completed full testing.

So, if your OpenStack system is at focal-ussuri, (today) the best thing to do is to use the latest/stable charms to OpenStack upgrade to wallaby, and then:

* Switch to xena/stable and upgrade to xena.
* Switch to yoga/stable and upgrade to yoga.

The current latest/stable charms do support xena, and thus *should* get the system from ussuri to wallaby (via victoria) - but, please, if you do have a lab, do test this; it has been tested previously, but obviously, all systems are different and it's hard to say it will be flawless.

Also note, that the ceph charms (ceph-mon, -osd, -cephfs, etc), have a slightly different naming convention (quincy, pacific, nautilus, etc) and thus change from the 21.10 charms, and OVN also has differently named tracks. Please see https://docs.openstack.org/charm-guide/latest/project/charm-delivery.html for more details.

If you ever want to just ask questions, etc, then please feel free to reach us as IRC: OFTC, channel #openstack-charms, or on the discourse.charmhub.io using the 'openstack' tag.

[Alex Kavanagh (ajkavanagh)]

The bug is that layer:openstack-principle had an update to handle package upgrades. This introduced the breaking change; the charms.openstack at stable/victoria doesn't handle that new feature, but unfortunately the charms have been built with the wrong version of layer:openstack principle.

The fix is to change the build.lock to ensure that the layer is properly pinned to:
commit 8100de4b24ab361423363c75599124708169d871

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-aodh/+/858653

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-placement/+/858657

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-barbican/+/858658

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-designate/+/858661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-magnum/+/858665

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-cell-controller/+/858671

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/858672

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-aodh/+/858775

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/858776

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-placement/+/858777

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-barbican/+/858778

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-manila/+/858779

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-designate/+/858780

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-magnum/+/858782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-masakari/+/858783

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/858786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-gnocchi/+/858787

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-cell-controller/+/858788

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/858789

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-aodh/+/858910

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/858911

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-placement/+/859554

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-barbican/+/859555

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-manila/+/859556

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-designate/+/859557

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-magnum/+/859558

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-masakari/+/859559

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/859560

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-gnocchi/+/859562

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-cell-controller/+/859563

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/859564

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-aodh/+/858910
Committed: https://opendev.org/openstack/charm-aodh/commit/5899c99d38e4d2c6151c32ab050216ea4ecc9dfd
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 5899c99d38e4d2c6151c32ab050216ea4ecc9dfd
Author: Alex Kavanagh <email address hidden>
Date: Thu Sep 22 12:40:48 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I1172fa06b873179af63c89e19715455a579e75e9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-aodh/+/858653
Committed: https://opendev.org/openstack/charm-aodh/commit/b5a0ffb0f07c3963060a1b0591ee7e0b6d545037
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b5a0ffb0f07c3963060a1b0591ee7e0b6d545037
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 10:44:16 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead.

    Closes-Bug: #1990062
    Change-Id: I07a1ed37261d7ff3bc3239a6362342a84c5439e6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-aodh/+/858775
Committed: https://opendev.org/openstack/charm-aodh/commit/59eef4000e4bfd0c87d6908359e0d24ce8ba02c0
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 59eef4000e4bfd0c87d6908359e0d24ce8ba02c0
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 18:04:51 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ib9be1b86907cb3df82784942791fe7e1b2966500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-barbican/+/858778
Committed: https://opendev.org/openstack/charm-barbican/commit/94d68b63ecae8123729e244d0c2972f410dd527a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 94d68b63ecae8123729e244d0c2972f410dd527a
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 18:10:35 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I21ed5da5af58500fde567c1f1956c16fdd185f6b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-placement/+/858777
Committed: https://opendev.org/openstack/charm-placement/commit/49d9dc8ec2299f683796ab203ce0860d93aeaa5b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 49d9dc8ec2299f683796ab203ce0860d93aeaa5b
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 18:09:44 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I2eca1dabf6d462e3e77435f893744e143267c388

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-manila/+/859680
Committed: https://opendev.org/openstack/charm-manila/commit/38c23188fefccd29afcb0482ba8bef68975fc976
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 38c23188fefccd29afcb0482ba8bef68975fc976
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:41:21 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ia18e4b8b3fca0f7a57c0298884a376b5e0756976

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-designate/+/859681
Committed: https://opendev.org/openstack/charm-designate/commit/949f3451fe4dc30342670f85dc7b6147acdf800a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 949f3451fe4dc30342670f85dc7b6147acdf800a
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:42:04 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I503a56c647527b3936054aaf6e0a4f9cd6528508

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-magnum/+/859684
Committed: https://opendev.org/openstack/charm-magnum/commit/abea7ad9b03c5399adfa560796d5a5795072da17
Submitter: "Zuul (22348)"
Branch: stable/xena

commit abea7ad9b03c5399adfa560796d5a5795072da17
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:45:10 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ie95ed4c64f2ba82948225e22d4da3db6d485afe9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-aodh/+/859674
Committed: https://opendev.org/openstack/charm-aodh/commit/be97d78f38f301c68fb4430705e5bd284b03df2c
Submitter: "Zuul (22348)"
Branch: stable/xena

commit be97d78f38f301c68fb4430705e5bd284b03df2c
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:34:58 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I500a841c2c41606c741b7f32a3401c5932e68360

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cell-controller/+/859606
Committed: https://opendev.org/openstack/charm-nova-cell-controller/commit/aa10393eab26b1c5475434f12dbd43c05ef4e0d5
Submitter: "Zuul (22348)"
Branch: stable/train

commit aa10393eab26b1c5475434f12dbd43c05ef4e0d5
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:38:42 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ic892c8ec10ab869512b72dabc5fe5c28cd5e5b83

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/859607
Committed: https://opendev.org/openstack/charm-ironic-api/commit/0c598ca1c9971dc182a0c835dfa081bb5dc67ba9
Submitter: "Zuul (22348)"
Branch: stable/train

commit 0c598ca1c9971dc182a0c835dfa081bb5dc67ba9
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:39:14 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I8a95db7519576319d85d37524c705d6c70f9aa35

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/859604
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/29cfbc12038718fa54ce70f6f3991ca04ae0c985
Submitter: "Zuul (22348)"
Branch: stable/train

commit 29cfbc12038718fa54ce70f6f3991ca04ae0c985
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:37:47 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I28f2bb46dd12f24b0073e0ffb6bf6f8a70299041

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-designate/+/859601
Committed: https://opendev.org/openstack/charm-designate/commit/3a920f868c32acfd5b0e32ffbf3f936c26211e5d
Submitter: "Zuul (22348)"
Branch: stable/train

commit 3a920f868c32acfd5b0e32ffbf3f936c26211e5d
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:35:31 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I1ecd7119afdb6a3d9827c17104a0c311a1f8e625

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-placement/+/859599
Committed: https://opendev.org/openstack/charm-placement/commit/aeb42127a220b1f6803f25b02e95185dbfabde1b
Submitter: "Zuul (22348)"
Branch: stable/train

commit aeb42127a220b1f6803f25b02e95185dbfabde1b
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:33:39 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I694b88590e126c498156cac54000767e3f5b0d4a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-gnocchi/+/859605
Committed: https://opendev.org/openstack/charm-gnocchi/commit/28838d70cfce320a8dd16418f543e282bc07f7b0
Submitter: "Zuul (22348)"
Branch: stable/train

commit 28838d70cfce320a8dd16418f543e282bc07f7b0
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:38:09 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Idb2b41b3e0ddcef77c301d82d5d383c5cb9b1c30

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-barbican/+/859600
Committed: https://opendev.org/openstack/charm-barbican/commit/bd8b42b0bfb3bb1ca0ebeeb900159b8f5ac516b5
Submitter: "Zuul (22348)"
Branch: stable/train

commit bd8b42b0bfb3bb1ca0ebeeb900159b8f5ac516b5
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:34:11 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I67edd2061821b87a49822420e77747abf0edc493

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-masakari/+/859603
Committed: https://opendev.org/openstack/charm-masakari/commit/32d5012d13326167f0a0a6c8bd38afe8e693624f
Submitter: "Zuul (22348)"
Branch: stable/train

commit 32d5012d13326167f0a0a6c8bd38afe8e693624f
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:37:21 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Iac844aba62453a88f5973046dd7681a8c615f206

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-aodh/+/859597
Committed: https://opendev.org/openstack/charm-aodh/commit/84b8b3880dff30df5041e419d4962ba92a1cd7b8
Submitter: "Zuul (22348)"
Branch: stable/train

commit 84b8b3880dff30df5041e419d4962ba92a1cd7b8
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 14:32:34 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I622ba45258c87b8ec25e2aaf5f522fd894dfb220

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-barbican/+/859679
Committed: https://opendev.org/openstack/charm-barbican/commit/e01b5ac9ab7feb83c6e593aadb1635a692c8366a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit e01b5ac9ab7feb83c6e593aadb1635a692c8366a
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:40:53 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I180b9b9f5861bce297ac77a4a416b1356c5a468e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/859675
Committed: https://opendev.org/openstack/charm-manila-ganesha/commit/74eb8bbc6d70af37b7f1b92f087ea010df4afba4
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 74eb8bbc6d70af37b7f1b92f087ea010df4afba4
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:35:31 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ia7cbd8e52dcca4cff62fa76a6497b01ae9e63242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-placement/+/859676
Committed: https://opendev.org/openstack/charm-placement/commit/cb2095d91df2ae619f29c05dafd1e16aaadfd176
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cb2095d91df2ae619f29c05dafd1e16aaadfd176
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:36:11 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I5d5b8a1044e4151fe502ffa94de32a145377511f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-designate/+/859557
Committed: https://opendev.org/openstack/charm-designate/commit/21c48c033a79b09e6cc6e198ac10afcd36421aec
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 21c48c033a79b09e6cc6e198ac10afcd36421aec
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 12:44:47 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Pin importlib-* for python 3.7 and 3.6

    Closes-Bug: #1990062
    Change-Id: I35077abad85e493a257b4e4ae396ffbb90238d69

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Felipe Reyes <email address hidden>" on branch: stable/21.10
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/859598
Reason: 21.10 is EOL

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Felipe Reyes <email address hidden>" on branch: stable/21.10
Review: https://review.opendev.org/c/openstack/charm-magnum/+/859602
Reason: 21.10 is EOL

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-masakari/+/858666
Committed: https://opendev.org/openstack/charm-masakari/commit/53b292abc984e167b39e5e5f3d1a00fcaff921ba
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 53b292abc984e167b39e5e5f3d1a00fcaff921ba
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 11:15:43 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead.

    Closes-Bug: #1990062
    Change-Id: I80944be90c922fb4abba9335fa49cff90c488333

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/859686
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/5401cd8d08cebeacac9ea15fae69a785a3f4c035
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5401cd8d08cebeacac9ea15fae69a785a3f4c035
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 19:45:52 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: Ib41edf3a27f671e89f120a945ff9258de6e21196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-magnum/+/858782
Committed: https://opendev.org/openstack/charm-magnum/commit/d877721e0560d2730f44e8049b2563c5b3e52701
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d877721e0560d2730f44e8049b2563c5b3e52701
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 18:15:49 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I87b69de7188d54cfde92b419a6031a9d34b92879

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-gnocchi/+/858787
Committed: https://opendev.org/openstack/charm-gnocchi/commit/49f4e998074a1c21b34db60ef647a2b8918de61d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 49f4e998074a1c21b34db60ef647a2b8918de61d
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 21 18:45:44 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I3493b03c1cbbb680f00919523ad887130487eb6f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-manila/+/859556
Committed: https://opendev.org/openstack/charm-manila/commit/f3f0c6a931900bebfa8c63880ffbe842ab337818
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f3f0c6a931900bebfa8c63880ffbe842ab337818
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 12:44:15 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I19f743b38e7aa63faf779a728eaee2bb55a18a8c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-masakari/+/859559
Committed: https://opendev.org/openstack/charm-masakari/commit/8c361b8257aed1b42cbfeefb37aa303a080f6f3f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8c361b8257aed1b42cbfeefb37aa303a080f6f3f
Author: Alex Kavanagh <email address hidden>
Date: Wed Sep 28 12:47:21 2022 +0100

    Pin build.lock for all layers to the commit hash

    The build.lock branch points to master which means that the layer is not
    locked. Lock the layers to the commit hash instead; this ensures
    reproducible builds as much as possible.

    Closes-Bug: #1990062
    Change-Id: I2a66ba368c068e333f6e5b131af6934cc3e31db8

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-ironic-conductor

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-ironic-api

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-gnocchi

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-aodh

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-barbican

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-designate

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-magnum

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-manila

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-manila-ganesha

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-masakari

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-nova-cell-controller

[Felipe Reyes (freyes)]

https://review.opendev.org/q/topic:bug/1990062+project:openstack/charm-placement

[Marcin Wilk (wilkmarcin)]

I am able to reproduce the problem when upgrading designate xena->yoga (current stable channel charms)
Initial designate charm: xena/stable 208
Target designate charm: yoga/stable 204

Unless I am missing something here, it looks to me the patch [1] hasn't been applied to the yoga/stable branch [2]. I checked the aodh yoga/stable charm code [3] and it looks similar. I suspect that other charms from the list above may be also affected.

[1] https://review.opendev.org/c/openstack/charm-designate/+/859681/1/src/build.lock
[2] https://opendev.org/openstack/charm-designate/src/branch/stable/yoga/src/build.lock
[3] https://opendev.org/openstack/charm-aodh/src/branch/stable/yoga/src/build.lock

Full console log is below (I did successful upgrade of the keystone charm first, followed by the failed designate upgrade):

# initial state before upgrade xena->yoga

juju status
Model Controller Cloud/Region Version SLA Timestamp
desig-test juju-maas maas/default 2.9.44 unsupported 10:05:02+01:00

App Version Status Scale Charm Channel Rev Exposed Message
designate 13.0.2 active 1 designate xena/stable 208 no Unit is ready
designate-mysql-router 8.0.35 active 1 mysql-router 8.0/stable 111 no Unit is ready
keystone 20.0.1+git20... active 1 keystone xena/stable 673 no Application Ready
keystone-mysql-router 8.0.35 active 1 mysql-router 8.0/stable 111 no Unit is ready
memcached active 1 memcached latest/stable 39 no Unit is ready
mysql 8.0.35 active 3 mysql-innodb-cluster 8.0/stable 107 no Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
rabbitmq-server 3.8.2 active 1 rabbitmq-server 3.8/stable 176 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
designate/0* active idle 0/lxd/0 10.10.10.94 9001/tcp Unit is ready
  designate-mysql-router/0* active idle 10.10.10.94 Unit is ready
keystone/0* active idle 0/lxd/1 10.10.10.95 5000/tcp Unit is ready
  keystone-mysql-router/0* active idle 10.10.10.95 Unit is ready
memcached/0* active idle 0/lxd/2 10.10.10.93 11211/tcp Unit is ready
mysql/0* active idle 0/lxd/3 10.10.10.92 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql/1 active idle 0/lxd/4 10.10.10.96 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql/2 active idle 0/lxd/5 10.10.10.98 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
rabbitmq-server/0* active idle 0/lxd/6 10.10.10.97 5672/tcp Unit i...

[Alex Kavanagh (ajkavanagh)]

@wilkmarcin

The bug in designate is different to this one. On the stable/yoga branch the designate charm has:

class DesignateCharmQueens(DesignateCharm):

    # This charms support Queens and onward
    release = 'queens'

    ...

    def run_upgrade(self, interfaces_list=None):
        """Upgrade OpenStack if an upgrade is available and action-managed
           upgrades is not enabled.
        :param interfaces_list: List of instances of interface classes
        :returns: None
        """
        super(DesignateCharmQueens, self).run_upgrade(
            interfaces_list=interfaces_list)
        memcached = relations.endpoint_from_flag(
            'coordinator-memcached.available')
        memcached.request_restart()

However, the action calls run_upgrade(...) as:

def openstack_upgrade_action(*args):
    """Run the openstack-upgrade action."""
    with charms_openstack.charm.provide_charm_instance() as charm_instance:
        charm_instance.run_upgrade(upgrade_openstack=True)
        charm_instance._assess_status()

i.e. with the "upgrade_openstack=True" param, and the charms.openstack lib also has that parameter.

This is an actual bug in the designate charm not being updated to support the "upgrade_openstack" optional param but it existing in the lib. As such it's a new/different bug.

[Marcin Wilk (wilkmarcin)]

Thank you @ajkavanagh for the verification.
I just opened a new lp2046449 for that.

[1] https://bugs.launchpad.net/charm-designate/+bug/2046449