CMR relation goes into terminated state on deploy

[Liam Young]

When deploying to two models with cmrs using bundles, the relations go onto a terminated state. This deployment was working with juju 2.7.0 but appears to be broken with juju 2.7.1

To reproduce see https://paste.ubuntu.com/p/FGjhwrt7Hj/ or below:

* Bundles *

$ cat swift-ny.yaml
series: bionic
applications:
  swift-proxy-region1:
    charm: cs:~gnuoy/swift-proxy-12
    num_units: 1
    options:
      region: RegionOne
      zone-assignment: manual
      replicas: 2
      enable-multi-region: true
      swift-hash: "global-cluster"
      read-affinity: "r1=100, r2=200"
      write-affinity: "r1, r2"
      write-affinity-node-count: '1'
      openstack-origin: cloud:bionic-train
  percona-cluster:
    charm: cs:~openstack-charmers-next/percona-cluster
    num_units: 1
    options:
      dataset-size: 25%
      max-connections: 1000
      source: cloud:bionic-train
  keystone:
    expose: True
    charm: cs:~openstack-charmers-next/keystone
    num_units: 1
    options:
      openstack-origin: cloud:bionic-train
relations:
  - - keystone:shared-db
    - percona-cluster:shared-db
  - - swift-proxy-region1:identity-service
    - keystone:identity-service

$ cat swift-ny-offers.yaml
applications:
  keystone:
    offers:
      keystone-offer:
        endpoints:
        - identity-service

$ cat swift-sf.yaml
series: bionic
applications:
  swift-proxy-region2:
    charm: cs:~gnuoy/swift-proxy-12
    num_units: 1
    options:
      region: RegionTwo
      zone-assignment: manual
      replicas: 2
      enable-multi-region: true
      swift-hash: "global-cluster"
      read-affinity: "r1=100, r2=200"
      write-affinity: "r1, r2"
      write-affinity-node-count: '1'
      openstack-origin: cloud:bionic-train

$ cat swift-sf-consumer.yaml
relations:
- - swift-proxy-region2:identity-service
  - keystone:identity-service
saas:
  keystone:
    url: admin/swift-ny.keystone-offer

* Deployment *

$ juju add-model swift-ny
Added 'swift-ny' model on serverstack/serverstack with credential 'gnuoy' for user 'admin'

$ juju add-model swift-sf
Added 'swift-sf' model on serverstack/serverstack with credential 'gnuoy' for user 'admin'

$ juju deploy -m swift-ny ./swift-ny.yaml --overlay ./swift-ny-offers.yaml
Resolving charm: cs:~openstack-charmers-next/keystone
Resolving charm: cs:~openstack-charmers-next/percona-cluster
Resolving charm: cs:~gnuoy/swift-proxy-12
Executing changes:
- upload charm cs:~openstack-charmers-next/keystone-475 for series bionic
- deploy application keystone on bionic using cs:~openstack-charmers-next/keystone-475
  added resource policyd-override
- expose keystone
- upload charm cs:~openstack-charmers-next/percona-cluster-359 for series bionic
- deploy application percona-cluster on bionic using cs:~openstack-charmers-next/percona-cluster-359
- upload charm cs:~gnuoy/swift-proxy-12 for series bionic
- deploy application swift-proxy-region1 on bionic using cs:~gnuoy/swift-proxy-12
- create offer keystone-offer using keystone:identity-service
- add relation keystone:shared-db - percona-cluster:shared-db
- add relation swift-proxy-region1:identity-service - keystone:identity-service
- add unit keystone/0 to new machine 0
- add unit percona-cluster/0 to new machine 1
- add unit swift-proxy-region1/0 to new machine 2
Deploy of bundle completed.

$ juju status -m swift-ny
Model Controller Cloud/Region Version SLA Timestamp
swift-ny gnuoy-serverstack serverstack/serverstack 2.7.1 unsupported 14:43:34Z

App Version Status Scale Charm Store Rev OS Notes
keystone 16.0.0 active 1 keystone jujucharms 475 ubuntu exposed
7percona-cluster 5.7.20 active 1 percona-cluster jujucharms 359 ubuntu
swift-proxy-region1 2.23.1 blocked 1 swift-proxy jujucharms 12 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0 172.20.0.11 5000/tcp Unit is ready
percona-cluster/0* active idle 1 172.20.0.42 3306/tcp Unit is ready
swift-proxy-region1/0* blocked idle 2 172.20.0.51 8080/tcp Missing relation: storage

Machine State DNS Inst id Series AZ Message
0 started 172.20.0.11 fac3da44-99f2-4f26-bef2-5cc26759fd45 bionic nova ACTIVE
1 started 172.20.0.42 dba27f3c-c673-4cde-bd83-104c441ac9d9 bionic nova ACTIVE
2 started 172.20.0.51 a8d648cb-94e0-443e-905d-116e395e6c7c bionic nova ACTIVE

Offer Application Charm Rev Connected Endpoint Interface Role
keystone-offer keystone keystone 475 0/0 identity-service keystone provider

$ juju deploy -m swift-sf ./swift-sf.yaml --overlay ./swift-sf-consumer.yaml
Resolving charm: cs:~gnuoy/swift-proxy-12
Executing changes:
- upload charm cs:~gnuoy/swift-proxy-12 for series bionic
- deploy application swift-proxy-region2 on bionic using cs:~gnuoy/swift-proxy-12
- consume offer keystone at admin/swift-ny.keystone-offer
Added gnuoy-serverstack:admin/swift-ny.keystone-offer as keystone
- add relation swift-proxy-region2:identity-service - keystone:identity-service
- add unit swift-proxy-region2/0 to new machine 0
Deploy of bundle completed.

$ juju status -m swift-sf
Model Controller Cloud/Region Version SLA Timestamp
swift-sf gnuoy-serverstack serverstack/serverstack 2.7.1 unsupported 14:44:29Z

SAAS Status Store URL
keystone terminated gnuoy-serverstack admin/swift-ny.keystone-offer

App Version Status Scale Charm Store Rev OS Notes
swift-proxy-region2 waiting 0/1 swift-proxy jujucharms 12 ubuntu

Unit Workload Agent Machine Public address Ports Message
swift-proxy-region2/0 waiting allocating 0 waiting for machine

Machine State DNS Inst id Series AZ Message
0 pending pending bionic instance "d8911474-a838-469a-99f3-c3d62a995088" has status BUILD, wait 10 seconds before retry, attempt 2

$ juju status -m swift-sf
Model Controller Cloud/Region Version SLA Timestamp
swift-sf gnuoy-serverstack serverstack/serverstack 2.7.1 unsupported 14:47:49Z

SAAS Status Store URL
keystone terminated gnuoy-serverstack admin/swift-ny.keystone-offer

App Version Status Scale Charm Store Rev OS Notes
swift-proxy-region2 2.23.1 blocked 1 swift-proxy jujucharms 12 ubuntu

Unit Workload Agent Machine Public address Ports Message
swift-proxy-region2/0* blocked idle 0 172.20.0.36 8080/tcp Missing relation: storage

Machine State DNS Inst id Series AZ Message
0 started 172.20.0.36 d8911474-a838-469a-99f3-c3d62a995088 bionic nova ACTIVE

[Liam Young (gnuoy)]

I switch back to 2.7.0 and reran the above. The consumer line in juju status never goes into a 'terminated' state, it stays 'active'. So this definitely looks like it was introduced by 2.7.1

[Simon Richardson (simonrichardson)]

Getting an error in remote application worker. Investingating why that error exists.

[Liam Young (gnuoy)]

The issue appears to exist in juju 2.8 ( 2.8-beta1+develop-ec9b3e7 ) too

[Simon Richardson (simonrichardson)]

As a workaround for now:

 - Remove the SAAS block from swift-sf
 - Consume the offer from swift-ny

[Christian Muirhead (2-xtian)]

Ok, I've worked out what's happening here - it's because the offer name and application name are different. The code that registers the relation exports the offer name, but in WatchRelationChanges we incorrectly use the application name to find the token, this succeeds when they happen to be named the same but not in this case.

I'm fixing that now, but in the meantime the workaround is to rename the offer to keystone in the two overlays.

[Christian Muirhead (2-xtian)]

I'll change the integration tests we have to use different application and offer names as well.

[Liam Young (gnuoy)]

The workaround to make the offer name match the app name works for me, thank you.

[Christian Muirhead (2-xtian)]

PR for fix: https://github.com/juju/juju/pull/11160