All rules deleted from juju security groups on OpenStack

[Martijn van der Woud]

We have a juju installation on OpenStack, running the Kubernetes Core bundle with the Openstack-integrator charm. We are using Cinder volumes and Neutron loadbalancers. After running without any issue for several weeks, we have seen that the security groups created by Juju suddenly become empty: all rules are gone.

We have encountered this issue several times now, but we were not able to discover any pattern in when it is occurring, nor do we have any clue what could be causing this behavior. We do suspect an issue with Juju, since we have been running some other projects (without Juju) on the same Openstack cloud for years and we have never seen this behavior before.

Is there any logging we can check or debug settings that we can activate so that we can provide additional information here?

[Heather Lanigan (hmlanigan)]

Hi Martijn, what version of juju are you running? Which OpenStack release?

Check the juju.provider.openstack logging for more info:
`juju debug-log -m controller --replay --include-module juju.provider.openstack`

You may need to increase the logging level for the future, by adding `;juju.provider.openstack=DEBUG` to the model config logging-config.

[Martijn van der Woud (martijnvanderwoud)]

Hi Heather,

We are running juju version 2.8.3.

OpenStack is version 3, I believe. I have asked the support desk of our hosting provider to tell me the exact version and will add that info here when I receive it.

The debug logs just give me this:

```
machine-0: 17:57:29 INFO juju.provider.openstack opening model "controller"
machine-0: 17:57:30 INFO juju.provider.openstack opening model "controller"
machine-0: 17:57:30 INFO juju.provider.openstack opening model "controller"
```

I have added `;juju.provider.openstack=DEBUG` to the logging-config, so hopefully we will be able to gather more useful information later

Also: yesterday I saw security rules being wiped immediately after deleting a K8S service of type LoadBalancer. I believe the OpenStack integrator charm is responsible for deletion of the neutron lbaas at that point, so maybe the problem could be in the integrator charm?

[Martijn van der Woud (martijnvanderwoud)]

Just heard back from my Cloud provider: OpenStack version is 2.0 Okata

[Heather Lanigan (hmlanigan)]

Martijn, are you using ipv4, ipv6 or both?

[Martijn van der Woud (martijnvanderwoud)]

Hi Heather, we are using ipv4 only at the moment

[Pen Gale (pengale)]

I'm not sure that we have all the information that we need to reproduce. This is a worrying issue, though. Triaging as high and dropping it into a near future milestone to remind us to follow-up, and also to try to reproduce on our end.

[Launchpad Janitor (janitor)]

[Expired for Canonical Juju because there has been no activity for 60 days.]