ovn-central inconsistently blocks after vault:certificates relation

Bug #2044324 reported by Max Asnaashari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
charm-ovn-central
New
Undecided
Unassigned

Bug Description

Seemingly randomly, using the LXD Juju cloud provider with Juju 3.1 to 3.3, running
```
 juju deploy vault vault --model lxd-cloud/cell0-ovn-central --config auto-generate-root-ca-cert=true --config totally-unsecure-auto-unlock=true --channel 1.8/stable --num-units 1

juju deploy ovn-central ovn-central --model lxd-cloud/cell0-ovn-central --channel 23.03/stable --num-units 3

juju integrate ovn-central:certificates vault:certificates --model lxd-cloud/cell0-ovn-central
```

The ovn-central charm will get stuck in a blocked state, reporting ` Ports which should be open, but are not: 6641, 6642`

Juju debug-log shows the following:

```
 unit-ovn-central-46: 04:32:23 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined ERROR could not download resource: HTTP request failed: Get https://[fd42:7768:dbe4:bff0:216:3eff:fe18:db73]:17070/model/fa159398-1d3d-4c62-8939-58ee4b5b1c07/units/unit-ovn-central-46/resources/prometheus-ovn-exporter: resource#ovn-central/prometheus-ovn-exporter not found
unit-ovn-central-47: 04:32:26 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Failed to start ovn-central.service: Unit ovn-central.service is masked.
unit-ovn-central-47: 04:32:26 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Failed to start ovn-ovsdb-server-nb.service: Unit ovn-ovsdb-server-nb.service is masked.
unit-ovn-central-47: 04:32:26 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Failed to start ovn-ovsdb-server-sb.service: Unit ovn-ovsdb-server-sb.service is masked.
unit-ovn-central-47: 04:32:26 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Removed /etc/systemd/system/ovn-central.service.
unit-ovn-central-47: 04:32:27 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Created symlink /etc/systemd/system/multi-user.target.wants/ovn-central.service → /lib/systemd/system/ovn-central.service.
unit-ovn-central-47: 04:32:27 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Removed /etc/systemd/system/ovn-ovsdb-server-nb.service.
unit-ovn-central-47: 04:32:28 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Created symlink /etc/systemd/system/ovn-nb-ovsdb.service → /lib/systemd/system/ovn-ovsdb-server-nb.service.
unit-ovn-central-47: 04:32:28 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Removed /etc/systemd/system/ovn-ovsdb-server-sb.service.
unit-ovn-central-46: 04:32:28 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Failed to start ovn-central.service: Unit ovn-central.service is masked.
unit-ovn-central-46: 04:32:28 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Failed to start ovn-ovsdb-server-nb.service: Unit ovn-ovsdb-server-nb.service is masked.
unit-ovn-central-46: 04:32:28 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Failed to start ovn-ovsdb-server-sb.service: Unit ovn-ovsdb-server-sb.service is masked.
unit-ovn-central-47: 04:32:28 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed Created symlink /etc/systemd/system/ovn-sb-ovsdb.service → /lib/systemd/system/ovn-ovsdb-server-sb.service.
unit-ovn-central-46: 04:32:29 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Removed /etc/systemd/system/ovn-central.service.
unit-ovn-central-47: 04:32:29 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed 2023-11-23T04:32:29Z|00001|unixctl|WARN|failed to connect to /var/run/ovn/ovnsb_db.ctl
unit-ovn-central-47: 04:32:29 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed ovn-appctl: cannot connect to "/var/run/ovn/ovnsb_db.ctl" (No such file or directory)
unit-ovn-central-47: 04:32:29 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed 2023-11-23T04:32:29Z|00001|unixctl|WARN|failed to connect to /var/run/ovn/ovnsb_db.ctl
unit-ovn-central-47: 04:32:29 WARNING unit.ovn-central/47.ovsdb-peer-relation-changed ovn-appctl: cannot connect to "/var/run/ovn/ovnsb_db.ctl" (No such file or directory)
unit-ovn-central-46: 04:32:29 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Created symlink /etc/systemd/system/multi-user.target.wants/ovn-central.service → /lib/systemd/system/ovn-central.service.
unit-ovn-central-46: 04:32:30 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Removed /etc/systemd/system/ovn-ovsdb-server-nb.service.
unit-ovn-central-46: 04:32:30 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Created symlink /etc/systemd/system/ovn-nb-ovsdb.service → /lib/systemd/system/ovn-ovsdb-server-nb.service.
unit-ovn-central-46: 04:32:30 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Removed /etc/systemd/system/ovn-ovsdb-server-sb.service.
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined Created symlink /etc/systemd/system/ovn-sb-ovsdb.service → /lib/systemd/system/ovn-ovsdb-server-sb.service.
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined 2023-11-23T04:32:31Z|00001|unixctl|WARN|failed to connect to /var/run/ovn/ovnsb_db.ctl
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined ovn-appctl: cannot connect to "/var/run/ovn/ovnsb_db.ctl" (No such file or directory)
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined 2023-11-23T04:32:31Z|00001|unixctl|WARN|failed to connect to /var/run/ovn/ovnsb_db.ctl
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined ovn-appctl: cannot connect to "/var/run/ovn/ovnsb_db.ctl" (No such file or directory)
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined 2023-11-23T04:32:31Z|00001|unixctl|WARN|failed to connect to /var/run/ovn/ovnsb_db.ctl
unit-ovn-central-46: 04:32:31 WARNING unit.ovn-central/46.ovsdb-peer-relation-joined ovn-appctl: cannot connect to "/var/run/ovn/ovnsb_db.ctl" (No such file or directory)
unit-ovn-central-45: 04:32:31 WARNING unit.ovn-central/45.certificates-relation-changed ERROR could not download resource: HTTP request failed: Get https://10.184.200.233:17070/model/fa159398-1d3d-4c62-8939-58ee4b5b1c07/units/unit-ovn-central-45/resources/prometheus-ovn-exporter: resource#ovn-central/prometheus-ovn-exporter not found
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for internal
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for admin
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for public
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for internal
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for admin
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: No network binding for public
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: Skipping request for certificate for ip in internal space, no local address found
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: Skipping request for certificate for ip in admin space, no local address found
unit-ovn-central-45: 04:32:32 WARNING unit.ovn-central/45.juju-log certificates:85: Skipping request for certificate for ip in public space, no local address found
unit-ovn-central-46: 04:32:34 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for internal
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for admin
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for public
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for internal
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for admin
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: No network binding for public
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: Skipping request for certificate for ip in internal space, no local address found
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: Skipping request for certificate for ip in admin space, no local address found
unit-ovn-central-46: 04:32:35 WARNING unit.ovn-central/46.juju-log ovsdb-peer:83: Skipping request for certificate for ip in public space, no local address found
unit-ovn-central-47: 04:32:37 WARNING unit.ovn-central/47.certificates-relation-changed ERROR could not download resource: HTTP request failed: Get https://[fd42:7768:dbe4:bff0:216:3eff:fe18:db73]:17070/model/fa159398-1d3d-4c62-8939-58ee4b5b1c07/units/unit-ovn-central-47/resources/prometheus-ovn-exporter: resource#ovn-central/prometheus-ovn-exporter not found
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for internal
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for admin
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for public
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for internal
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for admin
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: No network binding for public
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: Skipping request for certificate for ip in internal space, no local address found
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: Skipping request for certificate for ip in admin space, no local address found
unit-ovn-central-47: 04:32:38 WARNING unit.ovn-central/47.juju-log certificates:85: Skipping request for certificate for ip in public space, no local address found
unit-ovn-central-46: 04:32:39 WARNING unit.ovn-central/46.ovsdb-peer-relation-changed ERROR could not download resource: HTTP request failed: Get https://[fd42:7768:dbe4:bff0:216:3eff:fe18:db73]:17070/model/fa159398-1d3d-4c62-8939-58ee4b5b1c07/units/unit-ovn-central-46/resources/prometheus-ovn-exporter: resource#ovn-central/prometheus-ovn-exporter not found
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for internal
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for admin
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for public
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for internal
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for admin
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: No network binding for public
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: Skipping request for certificate for ip in internal space, no local address found
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: Skipping request for certificate for ip in admin space, no local address found
unit-ovn-central-46: 04:32:42 WARNING unit.ovn-central/46.juju-log coordinator:84: Skipping request for certificate for ip in public space, no local address found
unit-ovn-central-46: 04:32:44 WARNING unit.ovn-central/46.coordinator-relation-changed ERROR could not download resource: HTTP request failed: Get https://[fd42:7768:dbe4:bff0:216:3eff:fe18:db73]:17070/model/fa159398-1d3d-4c62-8939-58ee4b5b1c07/units/unit-ovn-central-46/resources/prometheus-ovn-exporter: resource#ovn-central/prometheus-ovn-exporter not found

```

Tags: lxd-cloud
Max Asnaashari (masnax)
description: updated
summary: - ovn-central inconsistently blocks after vault:db relation
+ ovn-central inconsistently blocks after vault:certificates relation
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.