Cannot remove CMR offers

[Peter Sabaini]

I'm experiencing an issue with removing CMR offers.

Steps:

1. Create offer in maas-infra model

$ juju offer ud-ldap-server:udprovide

2. Add relationship in other model

$ juju switch openstack
$ juju add-relation ud-ldap-client:udconsume admin/maas-infra.ud-ldap-server

3. Remove relation
$ juju remove-relation ...

4. Observe: offer is still present in maas-infra model

5. Try to remove offer

$ juju switch maas-infra
$ juju remove-offer --force ud-ldap-server:udprovide -y --debug
16:52:27 INFO juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
16:52:27 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/9102/bin/juju", "remove-offer", "--force", "ud-ldap-server:udprovide", "-y", "--debug"}
16:52:27 INFO juju.juju api.go:67 connecting to API addresses: [10.17.101.44:17070 10.17.101.45:17070 10.17.101.46:17070]
16:52:27 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.17.101.46:17070/api"
16:52:27 INFO juju.api apiclient.go:624 connection established to "wss://10.17.101.46:17070/api"
16:52:27 DEBUG juju.api monitor.go:35 RPC connection died

6. Observe: offer is still present

$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
ud-ldap-server admin 8 joined udprovide udldap-userdata provider 216.83.155.78/32

Also attempted to remove-saas in the openstack model (juju remove-saas ud-ldap-server). The remove-saas command succeeded but I'd still see the offer in the maas-infra model.

Here are some traces from the controller:

https://private-fileshare.canonical.com/~sabaini/14c66249-504d-49d1-84a4-e2f9e1ce2af0/machine-0-2019-10-23T18-36-01.343.log.gz

Version info:

$ snap list | grep juju
juju 2.6.9 9102 stable canonical* classic

[Peter Sabaini (peter-sabaini)]

Possibly related
Bug #1768682
Bug #1840850

[Ian Booth (wallyworld)]

Can we please get a juju dump-model from both models as well?
What does juju show-status-log on the units from both models in the cross model relation show?
We'd also need debug logging on these packages (from both controllers if offer/consume in different controllers):

juju.apiserver.crossmodelrelations
juju.apiserver.common.crossmodel
juju.worker.remoterelations

The attached log file is huge as it goes way back to when jujud was bootstrapped as 2.4.7, so just the logs from 2.6.9 onwards would be great.

remove-saas or remove-relation won't cause the offer to be removed. Only remove-offer does that. So we need to figure out what's stopping that from working. Normally, it requires a clean shutdown of any related consumers but if that's not possible, --force is meant to use a big hammer.

[Peter Sabaini (peter-sabaini)]

Hey Ian,

I've re-run remove-offer:

$ juju remove-offer --force ud-ldap-server:udprovide -y --debug
11:35:44 INFO juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
11:35:44 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/9102/bin/juju", "remove-offer", "--force", "ud-ldap-server:udprovide", "-y", "--debug"}
11:35:44 INFO juju.juju api.go:67 connecting to API addresses: [10.17.101.45:17070 10.17.101.46:17070 10.17.101.44:17070]
11:35:44 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.17.101.45:17070/api"
11:35:44 INFO juju.api apiclient.go:624 connection established to "wss://10.17.101.45:17070/api"
11:35:44 INFO juju.juju api.go:303 API endpoints changed from [10.17.101.45:17070 10.17.101.44:17070 10.17.101.46:17070] to [10.17.101.45:17070 10.17.101.46:17070 10.17.101.44:17070]
11:35:44 DEBUG juju.api monitor.go:35 RPC connection died
11:35:44 INFO cmd supercommand.go:502 command finished

I've uploaded diagnostics (show-status-log, sanitized bundles, log excerpts from the controller and the units): https://private-fileshare.canonical.com/~sabaini/14c66249-504d-49d1-84a4-e2f9e1ce2af0/lp1849561.tar.gz

There is only one controller. Fwiw if it helps with filtering, Juju models were upgraded to 2.6.9 on 2019-10-17

cheers,
peter.

[Ian Booth (wallyworld)]

Thanks for all the info. We also need juju dump-model from each of the 2 models please.

[Peter Sabaini (peter-sabaini)]

Hey, uploaded them here: https://private-fileshare.canonical.com/~sabaini/14c66249-504d-49d1-84a4-e2f9e1ce2af0/lp1849561-dumps.tar.gz

[Peter Sabaini (peter-sabaini)]

I should maybe add that the ud-ldap-server units changed, I've removed /1 and added what became /4

[Peter Sabaini (peter-sabaini)]

Subscribing field-high as this is blocking us in prod

[Ian Booth (wallyworld)]

Looking at the model db dump of the maas-infra model, the offer is gone

The relevant application section in the yaml dump:

- applications
  - application-config:
      trust: false
    charm-mod-version: 2
    charm-url: local:bionic/userdir-ldap-4
    cs-channel: ""
    endpoint-bindings:
      general-info: ""
      udconsume: ""
      udprovide: ""
    leader: ud-ldap-server/4
    leadership-settings: {}
    name: ud-ldap-server
...

contains no offers section. It would have had something like this had the offer existed:

    offers:
      offers:
      - acl:
          admin: admin
          everyone@external: read
        endpoints:
        - udprovide
        offer-name: ud-ldap-server
      version: 1

What is still present in the offering model is the relation to that offer from the consuming side:

relations:
  - endpoints:
      endpoints:
      - application-name: ud-ldap-server
        interface: udldap-userdata
        name: udprovide
        role: provider
      - application-name: remote-b545715c2ca243138f0a5342bcc441b7
        interface: udldap-userdata
        name: udconsume
        role: requirer
      version: 1
    id: 8
    key: remote-b545715c2ca243138f0a5342bcc441b7:udconsume ud-ldap-server:udprovide

This relation can be removed with:

$ juju remove-relation 8 --force

Removing an in-use offer requires --force and this is expected to also removes any relations to that offer. Why that didn't happen hear is as yet unclear.

[Ian Booth (wallyworld)]

I think I can see what the issue is.

remove-offer takes the offer name as the argument.
It looks like attempts were made to remove the offer using name with the endpoint included.
remove-offer is idempotent so that it will succeed silently if the offer does not exist.

$ juju remove-offer ud-ldap-server:udprovide

is incorrect.

$ juju remove-offer ud-ldap-server

is correct. Or you can fully specify the target URL

$ juju remove-offer admin/maas-infra.ud-ldap-server

When an offer is made, you do need to specify what endpoint to offer on if there's more than one. But the name of the offer is the application name or an alias if one is provided.
And when you relate to the offer, you relate to the name (or a URL with the name).
The :endpoint bit is only relevant when the offer is created.

$ juju offer ud-ldap-server:udprovide
Application "ud-ldap-server" endpoints [udprovide] available at "admin/maas-infra.ud-ldap-server"

$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
ud-ldap-server -

You can see the offer is called "ud-ldap-server" at URL "admin/maas-infra.ud-ldap-server". This name or URL is what's needed when removing. If you are in the same model as the offer, just the name is needed.

Can you try again using the name rather than name:endpoint and confirm it works or not?

[Peter Sabaini (peter-sabaini)]

No dice I'm afraid --

$ juju switch maas-infra
foundations-maas:admin/openstack -> foundations-maas:admin/maas-infra

$ juju remove-offer --force ud-ldap-server -y --debug
07:18:37 INFO juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
07:18:37 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/9102/bin/juju", "remove-offer", "--force", "ud-ldap-server", "-y", "--debug"}
07:18:37 INFO juju.juju api.go:67 connecting to API addresses: [10.17.101.45:17070 10.17.101.46:17070 10.17.101.44:17070]
07:18:37 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.17.101.45:17070/api"
07:18:37 INFO juju.api apiclient.go:624 connection established to "wss://10.17.101.45:17070/api"
07:18:40 DEBUG juju.api monitor.go:35 RPC connection died
07:18:40 INFO cmd supercommand.go:502 command finished

$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
ud-ldap-server admin 8 joined udprovide udldap-userdata provider 216.83.155.78/32

$ juju remove-offer admin/maas-infra.ud-ldap-server --force -y

$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
ud-ldap-server admin 8 joined udprovide udldap-userdata provider 216.83.155.78/32

Thanks,
peter.

[Ian Booth (wallyworld)]

The offer is not there but the relation to the offer is.
As per comment #8, can you please try to remove relation 8.

Also note, when things are removed in Juju, they do not disappear straight away. Even with --force, Juju tries to cleanly remove the application associated with the offer. It needs to remove any relations and wait for those to run hooks etc. It needs to wait sufficient time for these things to complete. And if they don't then stuff is forcefully removed. We need to indicate that the wheels are spinning so to speak (we don't do that yet) so it's not obvious stuff is happening.

So remove the relation and see how it looks then. The yaml dump did not contain any offers.

[Peter Sabaini (peter-sabaini)]

Thanks, that seems to have helped. After juju remove-relation 8 --force I could now also remove the offer -- cheers!

[Peter Sabaini (peter-sabaini)]

Dropping ~field-high as we're unblocked

[Richard Harding (rharding)]

There's some things we could try to make removal more resilient but there's going to be a chunk of work to get there in the distributed nature of CMR setups. Keeping the bug around for the trick of force removing the relation.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.