juju deploy <bundle> twice removes the neutron ports from the openstack instances

[Felipe Reyes]

When deploying for second time a bundle to add an application that was recently removed, the neutron ports of the existing OpenStack instances associated to the model are removed leaving the instances without connectivity.

This was observed originally in a 19 machines model where the model was adding a single application (single unit) all the existing machines were left with no ports attached.

[Test Case]

juju 2.9.14 -> juju show-controller -> http://paste.ubuntu.com/p/8dxv27Bs6y/
Provider: openstack
Network: 1 network with 2 subnets, for example: https://pastebin.ubuntu.com/p/4d3pg4Ghqw/

1) Bootstrap a new controller on a network with 2 subnets.

juju bootstrap --no-gui --model-default network=$NETWORK --config network=$NETWORK --debug serverstack multisubnet
juju model-config -m controller logging-config='<root>=DEBUG'
juju model-config logging-config='<root>=DEBUG'

2) Create 2 spaces

juju add-space main 10.0.0.0/24
juju add-space secondary 10.10.0.0/24

3) Deploy a 2 applications bundle, each application is a single unit

cat << EOF > /tmp/my-bundle.yaml
series: bionic
applications:
  keystone:
    charm: cs:keystone
    num_units: 1
    bindings:
      "": main
  percona-cluster:
    constraints: "spaces=main,secondary"
    charm: cs:percona-cluster
    num_units: 1
    bindings:
      "": main
      cluster: secondary
relations:
- - keystone:shared-db
  - percona-cluster:shared-db
EOF
juju deploy /tmp/my-bundle.yaml
juju wait

4) Remove keystone application

juju remove-application keystone
juju wait

5) Re-deploy the same bundle
juju deploy /tmp/my-bundle.yaml

Expected result:

A new keystone application is deployed and related to percona-cluster.

Actual result:

A new keystone application is deployed and related to percona-cluster, but in the process the ports the neutron ports are removed from percona-cluster/0

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default multisubnet serverstack/serverstack 2.9.14 unsupported 13:15:02Z

App Version Status Scale Charm Store Channel Rev OS Message
keystone 13.0.4 waiting 1 keystone charmstore stable 326 ubuntu Allowed_units list provided but this unit not present
percona-cluster 5.7.20 active 0/1 percona-cluster charmstore stable 299 ubuntu Unit is ready

Unit Workload Agent Machine Public address Ports Message
keystone/1* blocked idle 2 10.0.0.24 5000/tcp Incomplete relations: database, Allowed_units list provided but this unit not present
percona-cluster/0 unknown lost 1 10.0.0.130 3306/tcp agent lost, see 'juju show-status-log percona-cluster/0'

Machine State DNS Inst id Series AZ Message
1 down 10.0.0.130 7f3035d0-604e-4063-8aff-3476f941e1ec bionic nova ACTIVE
2 started 10.0.0.24 ec50e9df-4161-4566-9c0c-0ac5bfcd5dcc bionic nova ACTIVE

$ openstack port list --server 7f3035d0-604e-4063-8aff-3476f941e1ec # percona-cluster/0 unit, empty list.

$ openstack port list --network blue_network
+--------------------------------------+-------------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+-------------+-------------------+----------------------------------------------------------------------------+--------+
| 26fc4b08-ad65-4440-a971-34f66d78740e | port-ironic | fa:16:3e:a2:f2:b3 | ip_address='10.10.0.106', subnet_id='e3e1be59-603b-4a4e-95c5-55476127894f' | DOWN |
| 3a939cb8-8e46-4427-9ec3-93b5b9a6787d | | fa:16:3e:d2:d7:f2 | ip_address='10.0.0.2', subnet_id='328f8f85-213a-4b33-a52b-d778f8ab500e' | ACTIVE |
| | | | ip_address='10.10.0.2', subnet_id='e3e1be59-603b-4a4e-95c5-55476127894f' | |
| 4a0e8ffc-84b6-453e-9e0e-f8651f53a8b8 | | fa:16:3e:02:7c:4e | ip_address='10.0.0.1', subnet_id='328f8f85-213a-4b33-a52b-d778f8ab500e' | ACTIVE |
| 740cf7f0-3afd-4207-beaf-cbbdab3df178 | port-main | fa:16:3e:cd:d5:84 | ip_address='10.0.0.57', subnet_id='328f8f85-213a-4b33-a52b-d778f8ab500e' | DOWN |
| be368655-d346-46b2-8c0a-041975552ab3 | | fa:16:3e:88:06:8f | ip_address='10.0.0.114', subnet_id='328f8f85-213a-4b33-a52b-d778f8ab500e' | ACTIVE |
| c1c17a4a-38d5-4636-9d6d-d1032e85091a | | fa:16:3e:3d:49:cd | ip_address='10.0.0.24', subnet_id='328f8f85-213a-4b33-a52b-d778f8ab500e' | ACTIVE |
+--------------------------------------+-------------+-------------------+----------------------------------------------------------------------------+--------+

[Felipe Reyes (freyes)]

Crashdump for controller model: private-fileshare.canonical.com/~freyes/juju-crashdump-c4b73f1b-4288-4cfc-90f5-4d4dcd1030e1.tar.xz

[Felipe Reyes (freyes)]

Crashdump for the 'default' model: https://private-fileshare.canonical.com/~freyes/juju-crashdump-241bfadc-0d32-45d5-bce6-2a7f295ced58.tar.xz

[John A Meinel (jameinel)]

To clarify, the keystone and percona-cluster in this scenario are not the keystone of the openstack that it is deploying onto, but a separate instance of keystone that is being tested?
What is the step that creates the ports on percona-cluster? There isn't any reason that a relation would be creating or removing ports. It is something where the charm is calling 'open-port' and 'close-port' in response to be related to another application (and having that relation removed when you remove the application)?

[John A Meinel (jameinel)]

Can you include the output of 'juju show-status-log percona-cluster/0' to see what hooks have been fired for that application, to make sure that we set up the right relation events.

[Felipe Reyes (freyes)]

On Tue, 2021-09-14 at 21:40 +0000, John A Meinel wrote:
> To clarify, the keystone and percona-cluster in this scenario are not
> the keystone of the openstack that it is deploying onto, but a
> separate instance of keystone that is being tested?

Correct. They are being deployed on top of openstack, a different pair
of application (e.g. wordpress + mysql) should produce the same
behavior.

> What is the step that creates the ports on percona-cluster?

juju did it for me due to `constraints: "spaces=main,secondary"`, I
didn't do anything outside juju.

> There isn't any reason that a relation would be creating or removing
> ports. It is something where the charm is calling 'open-port' and
> 'close-port' in response to be related to another application (and
> having that relation removed when you remove the application)?

I don't think so, I will test this using a different set of
applications to rule a charm's (mis)behavior out.

[Felipe Reyes (freyes)]

On Tue, 2021-09-14 at 21:42 +0000, John A Meinel wrote:
> Can you include the output of 'juju show-status-log percona-
> cluster/0'
> to see what hooks have been fired for that application, to make sure
> that we set up the right relation events.

I don't have that environment around anymore, I will capture this
output on the next test I will do.

[Felipe Reyes (freyes)]

I used a different bundle with memcached, nrpe and nagios
http://paste.ubuntu.com/p/4tW93NfhSt/

In this setup just removing the "memcached" application left nagios/0
without connectivity, see https://pastebin.ubuntu.com/p/SnZVtgCMrC/

This is the requested show-status-log for the unit that loses
connectivity:
$ juju show-status-log nagios/0
Time Type Status Message
15 Sep 2021 01:48:43Z juju-unit allocating
15 Sep 2021 01:48:43Z workload waiting waiting for machine
15 Sep 2021 01:51:54Z workload waiting installing agent
15 Sep 2021 01:51:54Z workload waiting agent initializing
15 Sep 2021 01:51:59Z workload maintenance installing charm
software
15 Sep 2021 01:51:59Z juju-unit executing running install hook
15 Sep 2021 01:51:59Z workload maintenance Installing
15 Sep 2021 01:53:26Z juju-unit executing running monitors-
relation-created hook
15 Sep 2021 01:53:26Z juju-unit executing running leader-elected
hook
15 Sep 2021 01:53:26Z juju-unit executing running config-changed
hook
15 Sep 2021 01:53:31Z juju-unit executing running start hook
15 Sep 2021 01:53:33Z workload active ready
15 Sep 2021 01:53:33Z juju-unit executing running monitors-
relation-joined hook for nrpe/0
15 Sep 2021 01:53:33Z juju-unit executing running monitors-
relation-changed hook for nrpe/0
15 Sep 2021 01:53:34Z juju-unit idle
15 Sep 2021 01:56:54Z juju-unit executing running monitors-
relation-departed hook for nrpe/0
15 Sep 2021 01:56:55Z juju-unit idle