Upgrade charm fails looking for handler that is in new charms.openstack code

[David Ames]

The fix for related bug LP Bug #1858132 [0] added a default handler for cluster.available.

Field experienced failed charm upgrades for Designate and Octavia in what appears to be a race condition in which the reactive handlers expect a default handler which has not yet been installed via wheelhouse.

In other words, the code in charms.openstack hasn't been unpacked before reactive is looking for that code.

The following Traceback occurs:

2020-04-27 17:15:00 DEBUG upgrade-charm Installing collected packages: PyYAML, pyaml, netaddr, Tempita, six, MarkupSafe, Jinja2, charmhelpers, charms.openstack, psutil, dnspython, netifaces, charms.reactive, wheel
2020-04-27 17:15:01 DEBUG upgrade-charm Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 PyYAML-5.2 Tempita-0.5.2 charmhelpers-0.20.10 charms.openstack-0.0.1.dev1 charms.reactive-1.3.0 dnspython-1.16.0 netaddr-0.7.19 netifaces-0.10.9 psutil-5.7.0 pyaml-20.4.0 six-1.14.0 wheel-0.33.6
2020-04-27 17:15:01 DEBUG worker.uniter.jujuc server.go:182 running hook tool "juju-log"
2020-04-27 17:15:01 INFO juju-log Reactive main running for hook upgrade-charm
2020-04-27 17:15:01 DEBUG worker.uniter.jujuc server.go:182 running hook tool "juju-log"
2020-04-27 17:15:01 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 71, in main
    bus.discover()
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 424, in discover
    _register_handlers_from_file(search_path, filepath)
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 454, in _register_handlers_from_file
    _load_module(root, filepath)
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 442, in _load_module
    return importlib.import_module(package + module)
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/lib/juju/agents/unit-designate-0/charm/reactive/designate_handlers.py", line 30, in <module>
    'cluster.available',
  File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/defaults.py", line 48, in use_defaults
    .format(state))
RuntimeError: Default handler for 'cluster.available' doesn't exist

2020-04-27 17:15:01 DEBUG upgrade-charm Traceback (most recent call last):
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/charm/hooks/upgrade-charm", line 22, in <module>
2020-04-27 17:15:01 DEBUG upgrade-charm main()
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 71, in main
2020-04-27 17:15:01 DEBUG upgrade-charm bus.discover()
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 424, in discover
2020-04-27 17:15:01 DEBUG upgrade-charm _register_handlers_from_file(search_path, filepath)
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 454, in _register_handlers_from_file
2020-04-27 17:15:01 DEBUG upgrade-charm _load_module(root, filepath)
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 442, in _load_module
2020-04-27 17:15:01 DEBUG upgrade-charm return importlib.import_module(package + module)
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
2020-04-27 17:15:01 DEBUG upgrade-charm return _bootstrap._gcd_import(name[level:], package, level)
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2020-04-27 17:15:01 DEBUG upgrade-charm File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/charm/reactive/designate_handlers.py", line 30, in <module>
2020-04-27 17:15:01 DEBUG upgrade-charm 'cluster.available',
2020-04-27 17:15:01 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-designate-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/defaults.py", line 48, in use_defaults
2020-04-27 17:15:01 DEBUG upgrade-charm .format(state))
2020-04-27 17:15:01 DEBUG upgrade-charm RuntimeError: Default handler for 'cluster.available' doesn't exist
2020-04-27 17:15:02 ERROR juju.worker.uniter.operation runhook.go:132 hook "upgrade-charm" failed: exit status 1

[0] https://bugs.launchpad.net/charm-gnocchi/+bug/1858132

[David Ames (thedac)]

Field hack workaround:

Remove cluster.available from the following in reactive/$CHARM_NAME_handlers.py

charm.use_defaults(
    'cluster.available',
)

Removing this will stop reactive from looking for the default handler which should allow for the upgrade-charm hook to complete thus unpacking the charms.openstack code it is looking for.

[Steven Parker (sbparke)]

This issue was detected in octavia, designate and aodh.

Fresh installation of octavia got around this issue.
We could probably do the same for aodh but designate has IPs that are published upstream.

Steven

From juju status on 20.02 charm upgrades.

aodh/0 error idle 9/lxd/0 10.55.193.26 8042/tcp hook failed: "upgrade-charm"
aodh/1 error idle 10/lxd/0 10.55.192.218 8042/tcp hook failed: "upgrade-charm"
aodh/2* error idle 11/lxd/0 10.55.193.21 8042/tcp hook failed: "upgrade-charm"
designate/0 error idle 12/lxd/0 10.55.193.61 9001/tcp hook failed: "upgrade-charm"
designate/1* error idle 13/lxd/0 10.55.192.243 9001/tcp hook failed: "upgrade-charm"
designate/2 error idle 14/lxd/0 10.55.192.226 9001/tcp hook failed: "upgrade-charm"

[David Ames (thedac)]

Confirmed the following workaround:

1) In the error state, the charms_openstack/charm/defaults.py make_default_cluster_available_handler is missing
2) Removing cluster.available from reactive/desginate_handlers.py and re-running the upgrade-charm hook succeeds
3) the charms_openstack/charm/defaults.py make_default_cluster_available_handler is now there
 3.a) You can re-add cluster.available leaving no diff
4) Hooks succeed and the charm looks healthy
 4.a) You can even re-re-run the upgrade-charm hook if you want

[David Ames (thedac)]

For future developers:

TRIAGE:

This may be a bug in layer-basic. The race is the dependent code not unpacked before it gets called.

Simple re-create steps:

charm pull cs:designate-38
cd designate
juju deploy ./tests/bundles/bionic_train.yaml
(wait for cloud to settle)

git clone https://github.com/openstack/charm-designate/
mv charm-designate designate
tox -e build
cd build/builds/designate
juju upgrade-charm --switch .

The charm will be in a hook error as described above.

[Chris Sanders (chris.sanders)]

Subscribing field critical, David is already working on this with the team but this qualifies as it appears to affect any environment trying to perform 20.02 upgrades.

[David Ames (thedac)]

Workaround update:

Although, a "juju resolved designate/0" does NOT resolve the issue.
Re-running the upgrade-charm hook in a debug-hooks session does WITHOUT the need to edit anything.

This may be a hint to what the root cause is.

[Steven Parker (sbparke)]

Ok so it looks like I simply run the following command to resolve.

juju debug-hooks designate/0 upgrade-charm

[Steven Parker (sbparke)]

Ok so this works:

   enter debug mode on unit
   juju debug-hooks app/unit

   in another window juju resolved app/unit
   once that error is resolved you will drop into the actual debug session

   run hooks/upgrade-charm in tmux session
   exit from tmux debug session

[Steven Parker (sbparke)]

gnocchi is also affected by this bug.

[David Ames (thedac)]

From Frode Nordahl
--quote--
When you bring it up I do have had some difficulties lately during local iterative charm development that involve updating dependencies (charm-helpers, charms.openstack).

The venv appear to no longer be updated with data from the new charm, at least not in the same way as before. Have not gotten to the bottom of it.

Perhaps this is a general regression from layer-basic or juju?

Does it make a difference if the charm is built with layer basic from before [0] ?

0: https://github.com/juju-solutions/layer-basic/commit/597bed6d22b93ed0754f3f02b632ebbcf8206596
--end quote--

This is looking more and more like a layer basic ordering bug during upgrade-charm. The dependencies are not being updated before reactive handler code is called.

Digging for more proof.

[David Ames (thedac)]

tl;dr This is a layer basic bug not an OpenStack charms bug

Per fnordahl's hint I built a new version of designate (with the charms.openstack changes) with layer-basic one commit before https://github.com/juju-solutions/layer-basic/commit/597bed6d22b93ed0754f3f02b632ebbcf8206596 and it upgrades without error.

That seems like pretty definitive proof this is a layer-basic bug.

For future tests:
From cs:designate-38 to cs:designate-41 shows the failure.

cs:~thedac/designate-0 has layer-basic @ commit 0c8d7e28aeb87394dca57e76593ffcb7cc936cfa and everything else up to date.

From cs:designate-38 to cs:~thedac/designate-0 successfully upgrades.

[David Ames (thedac)]

Layer basic github issue: https://github.com/juju-solutions/layer-basic/issues/166

OpenStack reactive charms will need to be rebuilt with whatever fix comes out of this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724157

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724158

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724159

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724160

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724161

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724162

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724163

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724164

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724165

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724166

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724168

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724169

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724170

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724171

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724172

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724173

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724174

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.02
Review: https://review.opendev.org/724175

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724162
Committed: https://git.openstack.org/cgit/openstack/charm-masakari/commit/?id=a3e02fbf9c642c249be4a3be28b9d549c73d6367
Submitter: Zuul
Branch: master

commit a3e02fbf9c642c249be4a3be28b9d549c73d6367
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:41 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I47b87a83a0fa79ce90f8a39e2eaf7523bc5c141e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724172
Committed: https://git.openstack.org/cgit/openstack/charm-masakari/commit/?id=635f1a4f308072f650621582e8d2cda1b3935ca2
Submitter: Zuul
Branch: stable/20.02

commit 635f1a4f308072f650621582e8d2cda1b3935ca2
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:18:06 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I4937150fc2d86bf9c8703ab19e749eb23e8b9889

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724159
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=0a3f0b9356b4a84e7b199f134b1ebbf4bb8e27ae
Submitter: Zuul
Branch: master

commit 0a3f0b9356b4a84e7b199f134b1ebbf4bb8e27ae
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:20 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: Iecffc54c72693437e88c4d3effaa5e3a6d6df20d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724163
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/commit/?id=e1ec06c5c69a0e8c9a02c0193ecbbf30e401450b
Submitter: Zuul
Branch: master

commit e1ec06c5c69a0e8c9a02c0193ecbbf30e401450b
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:48 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I5df34b0439cb92db558db0cc9b1bcf620318aae4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724157
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=d1813aae47a9565389fd05f6ed1ee036aa028004
Submitter: Zuul
Branch: master

commit d1813aae47a9565389fd05f6ed1ee036aa028004
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:06 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I50b3f65288f089633d83c424a71dc79017513c91

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724161
Committed: https://git.openstack.org/cgit/openstack/charm-manila/commit/?id=0ecc8ecaccf4cfbc76ad99eb6b36546b621d3f4f
Submitter: Zuul
Branch: master

commit 0ecc8ecaccf4cfbc76ad99eb6b36546b621d3f4f
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:35 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I3a5d2a8d65ac2eaaf007126156c7c6e023ad79df

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724158
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=7b30cea4861f22f132379a7510992ced086cb53f
Submitter: Zuul
Branch: master

commit 7b30cea4861f22f132379a7510992ced086cb53f
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:14 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I193337bbf53d0f5d396eee6b630ccc5d692b33cd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724170
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=cb52db643e4103f1678e7231178eef913810ee2d
Submitter: Zuul
Branch: stable/20.02

commit cb52db643e4103f1678e7231178eef913810ee2d
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:17:53 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I680f276e255ddf667ccc228acbd88576661c6d26

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724165
Committed: https://git.openstack.org/cgit/openstack/charm-placement/commit/?id=5febca80a36dedfb8b17e9422f340ba7dd15e1d0
Submitter: Zuul
Branch: master

commit 5febca80a36dedfb8b17e9422f340ba7dd15e1d0
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:17:01 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I675ae1120b86d21b2acb5243fdc14765987cfdc8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724160
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=ecda4fc95a99e6e7c823e3fdc01a6e7376f0c69e
Submitter: Zuul
Branch: master

commit ecda4fc95a99e6e7c823e3fdc01a6e7376f0c69e
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:28 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I04f49f9d013a35f6877416410fba428a0be35fea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724175
Committed: https://git.openstack.org/cgit/openstack/charm-placement/commit/?id=76488814bbbcdd52578d5c94ab42b352c3207496
Submitter: Zuul
Branch: stable/20.02

commit 76488814bbbcdd52578d5c94ab42b352c3207496
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:18:26 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: Id9834902fab5314caa5ba20ea63a1144a56281b8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724168
Committed: https://git.openstack.org/cgit/openstack/charm-barbican/commit/?id=a4538671c29830a1c86fd12110f17445e6ac8498
Submitter: Zuul
Branch: stable/20.02

commit a4538671c29830a1c86fd12110f17445e6ac8498
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:17:40 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I0798bd0bd7cafdf84fd4cc0a3e883bc69985ace4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724173
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/commit/?id=5c0bfd4021525b68c0819b83f427b9e988305c34
Submitter: Zuul
Branch: stable/20.02

commit 5c0bfd4021525b68c0819b83f427b9e988305c34
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:18:12 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: Ic9b9a150f51a43f09fa23ced76b02280fe4d2ee0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724171
Committed: https://git.openstack.org/cgit/openstack/charm-manila/commit/?id=f5d1f40e6a0b41b99ec5bbb9c302df3eef47063d
Submitter: Zuul
Branch: stable/20.02

commit f5d1f40e6a0b41b99ec5bbb9c302df3eef47063d
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:18:00 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I6513044b43a59671689a5852f75d6679adaa7077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724166
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=855a5330c76c860390c20e06a4ef1bc5c957e8f1
Submitter: Zuul
Branch: stable/20.02

commit 855a5330c76c860390c20e06a4ef1bc5c957e8f1
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:17:33 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I8dd84bf536de32fb20e1eedaffc3335167520805

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724169
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=122fb899ad9ccfdce82f4ae3e2e3fcd45857fb98
Submitter: Zuul
Branch: stable/20.02

commit 122fb899ad9ccfdce82f4ae3e2e3fcd45857fb98
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:17:47 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: Ib425255eae412dacfd87a15ceecb4248c58c6a4e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724174
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=af0af133a7f4ba4b5f1f8ece5d07dc6057c29f86
Submitter: Zuul
Branch: stable/20.02

commit af0af133a7f4ba4b5f1f8ece5d07dc6057c29f86
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:18:19 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I26eaf7ae50fe528b7ab0bf5182aa1ed82b97cb7e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724164
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=70d07363c4c099421849aba5cd6af4123ece5dd8
Submitter: Zuul
Branch: master

commit 70d07363c4c099421849aba5cd6af4123ece5dd8
Author: David Ames <email address hidden>
Date: Tue Apr 28 14:16:55 2020 -0700

    Rebuild reactive APIs for layer-basic regression

    Fix regression in layer-basic which failed to unpack new dependencies
    properly during the upgrade-charm hook.

    Github Issue: #166
    Closes Bug: #1875496

    Change-Id: I4ad87b47a412986827e3a88c68e28d6b7344db97