Upgrade from latest/edge to 8.0/stable is failing

[Márton Kiss]

The upgrade from latest/edge to 8.0/stable is failing, I experienced that in a customer environment. The current revisions for the channels are:

latest/edge: rev 108
8.0/stable: rev 111

I would expect the rev 108 -> rev 111 upgrade path working. The charm had been upgraded was initiated with the following command:

```
$ juju refresh --switch mysql-router --channel 8.0/stable keystone-mysql-router --force-units
```

The upgrade is failing with the following stack trace in charm logs:

```
unit-keystone-mysql-router-1: 16:16:00 INFO unit.keystone-mysql-router/1.juju-log Invoking reactive handler: reactive/layer_openstack.py:46:default_upgrade_charm
unit-keystone-mysql-router-1: 16:16:00 INFO unit.keystone-mysql-router/1.juju-log Invoking reactive handler: reactive/mysql_router_handlers.py:18:db_router_request
unit-keystone-mysql-router-1: 16:16:00 ERROR unit.keystone-mysql-router/1.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
TypeError: db_router_request() missing 1 required positional argument: 'db_router'

unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm Traceback (most recent call last):
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm File "/var/lib/juju/agents/unit-keystone-mysql-router-1/charm/hooks/upgrade-charm", line 22, in <module>
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm main()
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm bus.dispatch(restricted=restricted_mode)
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm _invoke(other_handlers)
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm handler.invoke()
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm File "/var/lib/juju/agents/unit-keystone-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm self._action(*args)
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm TypeError: db_router_request() missing 1 required positional argument: 'db_router'
unit-keystone-mysql-router-1: 16:16:00 ERROR juju.worker.uniter.operation hook "upgrade-charm" (via explicit, bespoke hook script) failed: exit status 1
unit-keystone-mysql-router-1: 16:16:00 INFO juju.worker.uniter awaiting error resolution for "upgrade-charm" hook
unit-keystone-mysql-router-1: 16:16:01 INFO juju.worker.uniter awaiting error resolution for "upgrade-charm" hook
unit-keystone-mysql-router-1: 16:16:06 INFO juju.worker.uniter awaiting error resolution for "upgrade-charm" hook
```

The upgrade failure is reproducible with the following very simple bundle (deployed with juju 3.1.5):

```
series: jammy
machines:
  "1":
    constraints: tags=foundation-nodes,compute
  "2":
    constraints: tags=foundation-nodes,compute
  "3":
    constraints: tags=foundation-nodes,compute
applications:
  hacluster-keystone:
    charm: hacluster
    channel: 2.4/stable
    revision: 131
    options:
      cluster_count: 3
    bindings:
      "": oam-space
  keystone:
    charm: keystone
    channel: yoga/stable
    revision: 669
    num_units: 3
    to:
    - lxd:1
    - lxd:2
    - lxd:3
    options:
      openstack-origin: distro
      preferred-api-version: 3
      region: RegionOne
      token-expiration: 86400
      vip: 172.27.52.29
    bindings:
      "": oam-space
  mysql-innodb-cluster:
    charm: mysql-innodb-cluster
    channel: 8.0/stable
    revision: 107
    num_units: 3
    to:
    - lxd:1
    - lxd:2
    - lxd:3
    options:
      enable-binlogs: true
      innodb-buffer-pool-size: 8G
      max-connections: 4000
      wait-timeout: 3600
    bindings:
      "": oam-space
  keystone-mysql-router:
    charm: mysql-router
    channel: latest/edge
    revision: 108
    bindings:
      "": oam-space
relations:
- [ "keystone-mysql-router:shared-db", "keystone:shared-db" ]
- [ "mysql-innodb-cluster:db-router", "keystone-mysql-router:db-router" ]
- [ "hacluster-keystone:ha", "keystone:ha" ]
```

[Alex Kavanagh (ajkavanagh)]

> latest/edge: rev 108
> 8.0/stable: rev 111

> I would expect the rev 108 -> rev 111 upgrade path working. The charm had been upgraded was initiated with the following command:

That's actually a downgrade. latest/edge is a different version of the charm to 8.0/stable, and is 'more recent' in terms of software.

May I ask why you were changing from latest/edge to 8.0/stable?

[Márton Kiss (marton-kiss)]

This was a result of a channel misconfiguration, the failing code looks pretty much the same for me in both revisions:

rev108/reactive/mysql_router_handlers.py (latest/edge):
```
@reactive.when('charm.installed')
@reactive.when('db-router.connected')
def db_router_request(db_router):
    """Send DB Router request to MySQL InnoDB Cluster.

    Using the db-router interface send connection request.

    :param db_router: DB-Router interface
    :type db_router_interface: MySQLRouterRequires object
    """
    with charm.provide_charm_instance() as instance:
        db_router.set_prefix(instance.db_prefix)
        db_router.configure_db_router(
            instance.db_router_user,
            instance.db_router_address,
            prefix=instance.db_prefix)
        # Reset on scale in
        db_router.set_or_clear_available()
        instance.assess_status()
```

and the rev111/reactive/mysql_router_handlers.py (8.0/stable)

```
@reactive.when('charm.installed')
@reactive.when('db-router.connected')
def db_router_request(db_router):
    """Send DB Router request to MySQL InnoDB Cluster.

    Using the db-router interface send connection request.

    :param db_router: DB-Router interface
    :type db_router_interface: MySQLRouterRequires object
    """
    with charm.provide_charm_instance() as instance:
        db_router.set_prefix(instance.db_prefix)
        db_router.configure_db_router(
            instance.db_router_user,
            instance.db_router_address,
            prefix=instance.db_prefix)
        # Reset on scale in
        db_router.set_or_clear_available()
        instance.assess_status()
```

Alex, can you help me show me the latest commits of rev108 and rev111 in the https://opendev.org/openstack/charm-mysql-router/ repository? Or those two versions are coming from different branches during charm release?

[Alex Kavanagh (ajkavanagh)]

So the bug output is this:

unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm self._action(*args)
unit-keystone-mysql-router-1: 16:16:00 WARNING unit.keystone-mysql-router/1.upgrade-charm TypeError: db_router_request() missing 1 required positional argument: 'db_router'

and the (joint) code in question is:

@reactive.when('charm.installed')
@reactive.when('db-router.connected')
def db_router_request(db_router):
    """Send DB Router request to MySQL InnoDB Cluster.

    Using the db-router interface send connection request.

    :param db_router: DB-Router interface
    :type db_router_interface: MySQLRouterRequires object
    """

So, basically, charms.reactive is calling db_router_request() without the db_router arg which is supposed to come from "@reactive.when('db-router.connected')"

So it looks like a bug in charms.reactive.

Typically, 'easy' fixes are to do:

@reactive.when('charm.installed')
@reactive.when('db-router.connected')
def db_router_request():
    db_router = reactive.relations.endpoint_from_flag('db-router.connected')
    if not db_router:
        return
    with charm.provide_charm_instance() as instance:

As to why it is doing this; I guess we'd need to look at charms.reactive. It could be due to either a race in the interface (https://github.com/juju-solutions/charms.reactive/issues/231#issuecomment-775325418) or something not quite right during the upgrade in the sqlite db: https://github.com/juju-solutions/charms.reactive/issues/221

Either way, it looks like the current pattern in the code isn't robust enough to work around the issue in charms.openstack.

Note: charms.reactive is unlikely to be updated, particularly when we can work around it.

re:

> Alex, can you help me show me the latest commits of rev108 and rev111 in the https://opendev.org/openstack/charm-mysql-router/ repository? Or those two versions are coming from different branches during charm release?

So latest/edge is build from the "master" branch, whereas 8.0/stable is built from the stable/jammy branch.

But I don't think this error is due to the mysql-router code, per se, but rather something odd going on in charms.reactive when doing the juju switch.

[Alex Kavanagh (ajkavanagh)]

Triage as medium as the fix is fairly simple, but ops are unlikely to deploy latest/edge and then downgrade to 8.0/stable; if this occurs on simple refresh on 8.0/stable then re-evaluate to high/critical.