Octavia unit fails to install octavia-driver-agent

[Sérgio Manso]

Octavia failed to create octavia-driver-agent.service on one unit in two fresh deployments.

I'm using cs:octavia-30 in a focal-ussuri-ovn deployment.

The unit only entered in block state after running the configure-resources action.

Unit Workload Agent Machine Public address Ports Message
octavia/0* blocked idle 0/lxd/17 10.0.0.152 9876/tcp Services not running that should be: octavia-driver-agent
  hacluster-octavia/0* active idle 10.0.0.152 Unit is ready and clustered
  logrotated/19 active idle 10.0.0.152 Unit is ready.
  octavia-mysql-router/0* active idle 10.0.0.152 Unit is ready
  octavia-ovn-chassis/0* active idle 10.0.0.152 Unit is ready
  public-policy-routing/11 active idle 10.0.0.152 Unit is ready
octavia/1 active idle 1/lxd/17 10.0.0.241 9876/tcp Unit is ready
  hacluster-octavia/1 active idle 10.0.0.241 Unit is ready and clustered
  logrotated/23 active idle 10.0.0.241 Unit is ready.
  octavia-mysql-router/1 active idle 10.0.0.241 Unit is ready
  octavia-ovn-chassis/1 active idle 10.0.0.241 Unit is ready
  public-policy-routing/15 active idle 10.0.0.241 Unit is ready
octavia/2 active idle 2/lxd/17 10.0.0.144 9876/tcp Unit is ready
  hacluster-octavia/2 active idle 10.0.0.144 Unit is ready and clustered
  logrotated/54 active idle 10.0.0.144 Unit is ready.
  octavia-mysql-router/2 active idle 10.0.0.144 Unit is ready
  octavia-ovn-chassis/2 active idle 10.0.0.144 Unit is ready
  public-policy-routing/42 active idle 10.0.0.144 Unit is ready

$ juju run -a octavia "systemctl status octavia-driver-agent"
- ReturnCode: 4
  Stderr: |
    Unit octavia-driver-agent.service could not be found.
  Stdout: ""
  UnitId: octavia/0
- Stdout: |
    ● octavia-driver-agent.service - OpenStack Octavia Driver Agent (octavia-driver-agent)
         Loaded: loaded (/lib/systemd/system/octavia-driver-agent.service; enabled; vendor preset: enabled)
         Active: active (running) since Wed 2021-02-24 14:15:26 UTC; 11min ago
       (...)
  UnitId: octavia/1
- Stdout: |
    ● octavia-driver-agent.service - OpenStack Octavia Driver Agent (octavia-driver-agent)
         Loaded: loaded (/lib/systemd/system/octavia-driver-agent.service; enabled; vendor preset: enabled)
         Active: active (running) since Wed 2021-02-24 14:15:26 UTC; 11min ago
       (...)
  UnitId: octavia/2

Seems that octavia/0 skipped the installation of the driver agent:
Logs:
(octavia/2)
2021-02-24 14:09:03 DEBUG ovsdb-subordinate-relation-changed Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
2021-02-24 14:09:03 DEBUG ovsdb-subordinate-relation-changed Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [838 kB]
2021-02-24 14:09:03 DEBUG ovsdb-subordinate-relation-changed Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.7 kB]
2021-02-24 14:09:03 DEBUG ovsdb-subordinate-relation-changed Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [745 kB]
2021-02-24 14:09:04 DEBUG ovsdb-subordinate-relation-changed Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [155 kB]
2021-02-24 14:09:04 DEBUG ovsdb-subordinate-relation-changed Get:9 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.6 kB]
2021-02-24 14:09:10 DEBUG ovsdb-subordinate-relation-changed Fetched 2096 kB in 1s (1786 kB/s)
2021-02-24 14:09:10 DEBUG ovsdb-subordinate-relation-changed Reading package lists...
2021-02-24 14:09:14 INFO juju-log ovsdb-subordinate:186: Installing ['octavia-driver-agent', 'python3-ovn-octavia-provider'] with options: ['--option=Dpkg::Options::=--force-confold']
2021-02-24 14:09:14 DEBUG ovsdb-subordinate-relation-changed Reading package lists...
2021-02-24 14:09:14 DEBUG ovsdb-subordinate-relation-changed Building dependency tree
(...agent installation)
2021-02-24 14:09:22 INFO juju-log ovsdb-subordinate:186: Running maybe_do_policyd_overrides
2021-02-24 14:09:22 DEBUG juju-log ovsdb-subordinate:186: Cleaning path: /etc/octavia/policy.d
2021-02-24 14:09:22 INFO juju-log ovsdb-subordinate:186: Adding user systemd-network to group octavia
2021-02-24 14:09:22 DEBUG ovsdb-subordinate-relation-changed Adding user systemd-network to group octavia

(octavia/0)
2021-02-24 14:09:19 DEBUG ovsdb-subordinate-relation-changed Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
2021-02-24 14:09:20 DEBUG ovsdb-subordinate-relation-changed Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [838 kB]
2021-02-24 14:09:20 DEBUG ovsdb-subordinate-relation-changed Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.7 kB]
2021-02-24 14:09:20 DEBUG ovsdb-subordinate-relation-changed Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [745 kB]
2021-02-24 14:09:20 DEBUG ovsdb-subordinate-relation-changed Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [155 kB]
2021-02-24 14:09:20 DEBUG ovsdb-subordinate-relation-changed Get:9 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.6 kB]
2021-02-24 14:09:23 DEBUG ovsdb-subordinate-relation-changed Fetched 2096 kB in 1s (2157 kB/s)
2021-02-24 14:09:24 DEBUG ovsdb-subordinate-relation-changed Reading package lists...
2021-02-24 14:09:27 INFO juju-log ovsdb-subordinate:186: Running maybe_do_policyd_overrides
2021-02-24 14:09:27 DEBUG juju-log ovsdb-subordinate:186: Cleaning path: /etc/octavia/policy.d
2021-02-24 14:09:27 INFO juju-log ovsdb-subordinate:186: Adding user systemd-network to group octavia
2021-02-24 14:09:27 DEBUG ovsdb-subordinate-relation-changed Adding user systemd-network to group octavia

Workaround:

Run install hook in the blocked unit: juju run -u octavia/0 "hooks/install"

This way I can get the agent to be installed
021-02-24 14:32:06 INFO juju-log Installing ['octavia-driver-agent', 'python3-ovn-octavia-provider'] with options: ['--option=Dpkg::Options::=--force-confold']

[Alex Kavanagh (ajkavanagh)]

Please, if possible, could you attach the full log for the octavia unit that failed to install the octavia driver. It would be good to know what the sequence of hooks was and why install didn't have the desired affect. If there is a crashdump for this model then please could that be attached instead?

[Sérgio Manso (sergiomanso)]

Attaching logs from the blocked octavia unit.
(The cloud had to be redeployed, I'll get a crashdump if I face it again.)

[Sérgio Manso (sergiomanso)]

I'm attaching a crashdump. I've faced this issue again
https://private-fileshare.canonical.com/~sergiomanso/juju-crashdump-0ca6da19-02c9-4304-99fd-5a503c845cec.tar.xz

[Alex Kavanagh (ajkavanagh)]

Okay, I can see what is going on:

1. In the `maybe_enable_ovn_driver` function in octavia_handers.py the `charm.octavia.enable-ovn-driver` flag is set if the ovsdb-subordinate relation is available (i.e. made).
2. In the UssuriOctaviaCharm class, in the __init__ method, the packages are extended if the flag is set.

The issue is that the __init__ method runs before the maybe_enable_ovn_driver function can be run, so it will only be on another hook that the install will be tried again. Sadly, if the ovsdb-subordinate is the last relation to be made, there isn't another non 'update-status' hook to fire and so it won't be enabled.

So, this bug is due to the update-status being gated against the handler that used to, via a side-effect of update-status, complete the installation of the relevant package.

The correct action is to not perform the package declaration in the __init__ method, but to do it later as part of the handler run.

[Alex Kavanagh (ajkavanagh)]

Review https://review.opendev.org/c/openstack/charm-octavia/+/778003 should address the issue.