Default relation limit of 1 prevents adding relations

[Xav Paice]

Some charms do not specify a limit for relations in metadata.yaml because there's no need to, it should be unlimited (within reason).

Since https://github.com/juju/juju/pull/11446, in 2.8.0, the limit: has been enforced, but most charms do not specify a limit at all. The default, it appears, is 1 rather than unlimited.

This breaks some desired behavior, e.g. for Cinder and cinder-ceph where we should be able to specify multiple relations to the cinder-ceph subordinate.

E.g.:

juju deploy cs:cinder
juju deploy cs:cinder-ceph cinder-ceph1
juju deploy cs:cinder-ceph cinder-ceph2
juju deploy cs:nova-compute

juju add-relation cinder cinder-ceph1 # works fine
juju add-relation cinder cinder-ceph2
ERROR cannot add relation "cinder:storage-backend cinder-ceph2:storage-backend": establishing a new relation for cinder:storage-backend would exceed its maximum relation limit of 1 (quota limit exceeded)

juju add-relation nova-compute cinder-ceph1 # works fine
juju add-relation nova-compute cinder-ceph2
ERROR cannot add relation "nova-compute:ceph-access cinder-ceph2:ceph-access": establishing a new relation for nova-compute:ceph-access would exceed its maximum relation limit of 1 (quota limit exceeded)

[Achilleas Anagnostopoulos (achilleasa)]

I just checked and can confirm that the metadata for all referenced charms does not specify any relation limit. We need to check whether the metadata parsing code in the charm repo is trying to be smart by forcing a limit of 1 when it's not defined.

In the meantime, passing '--force' to the deploy command should get you up and running while we investigate this.

[Achilleas Anagnostopoulos (achilleasa)]

I have not been able to reproduce this error.

I have tried running the above set on commands on:
- a fresh 2.8.0 controller (from snap)
- a fresh 2.7.7 controller that has been upgraded to the 2.8.0 snap (both controller and model)

I also tried to run all commands both at once as well as waiting for the charms to finish installing before running the relation commands. In all scenarios, no errors where produced and the juju status output looks as follows:

$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
default test-1 lxd-with-cache/default 2.8.0 unsupported 10:30:30+01:00

App Version Status Scale Charm Store Rev OS Notes
cinder 12.0.9 blocked 1 cinder jujucharms 303 ubuntu
cinder-ceph1 12.0.9 blocked 1 cinder-ceph jujucharms 256 ubuntu
cinder-ceph2 12.0.9 blocked 1 cinder-ceph jujucharms 256 ubuntu
nova-compute 17.0.12 blocked 1 nova-compute jujucharms 318 ubuntu

Unit Workload Agent Machine Public address Ports Message
cinder/0* blocked idle 0 10.55.158.202 8776/tcp Missing relations: database, identity, messaging
  cinder-ceph1/0* blocked idle 10.55.158.202 Missing relations: ceph
  cinder-ceph2/0* blocked idle 10.55.158.202 Missing relations: ceph
nova-compute/0* blocked idle 1 10.55.158.127 Missing relations: messaging, image

Machine State DNS Inst id Series AZ Message
0 started 10.55.158.202 juju-8d25f4-0 bionic Running
1 started 10.55.158.127 juju-8d25f4-1 bionic Running

Relation provider Requirer Interface Type Message
cinder-ceph1:ceph-access nova-compute:ceph-access cinder-ceph-key regular
cinder-ceph1:storage-backend cinder:storage-backend cinder-backend subordinate
cinder-ceph2:ceph-access nova-compute:ceph-access cinder-ceph-key regular
cinder-ceph2:storage-backend cinder:storage-backend cinder-backend subordinate
cinder:cluster cinder:cluster cinder-ha peer
nova-compute:compute-peer nova-compute:compute-peer nova peer

Can you please try to reproduce again on a new controller and let me know if you did something differently? For the time being, I will mark this as incomplete as additional information is required for a reproducer.

[Trent Lloyd (lathiat)]

Maybe it doesn't reproduce unless you deploy the other charms properly, e.g. a full openstack-base bundle?

[Chris MacNaughton (chris.macnaughton)]

I can reproduce this on a much smaller bundle, after deploying at 2.7.x and upgrading to 2.8+:

$ juju status
jModel Controller Cloud/Region Version SLA Timestamp Notes
controller olympus olympus 2.8.0 unsupported 14:03:30+02:00 upgrade available: 2.8.1

App Version Status Scale Charm Store Rev OS Notes
grafana active 1 grafana jujucharms 38 ubuntu
prometheus active 1 prometheus2 jujucharms 18 ubuntu
prometheus-snmp-exporter active 1 prometheus-snmp-exporter jujucharms 1 ubuntu
telegraf active 3 telegraf jujucharms 37 ubuntu
ubuntu 18.04 active 3 ubuntu jujucharms 15 ubuntu

Unit Workload Agent Machine Public address Ports Message
grafana/0* active idle 1/lxd/0 <redacted-IPv6-prefix>:1000:216:3eff:fe06:c1f8 3000/tcp Started grafana-server
prometheus-snmp-exporter/1* active idle 2/lxd/2 <redacted-IPv6-prefix>:1000:216:3eff:fe8d:d84 9116/tcp Ready
prometheus/0* active idle 0/lxd/0 <redacted-IPv6-prefix>:1000:216:3eff:fe05:8415 9090/tcp,12321/tcp Ready
ubuntu/0* active idle 0 10.0.4.203 ready
  telegraf/1 active idle 10.0.4.203 9103/tcp Monitoring ubuntu/0
ubuntu/1 active idle 1 10.0.4.26 ready
  telegraf/0* active idle 10.0.4.26 9103/tcp Monitoring ubuntu/1
ubuntu/2 active idle 2 10.0.4.25 ready
  telegraf/2 active idle 10.0.4.25 9103/tcp Monitoring ubuntu/2

Machine State DNS Inst id Series AZ Message
0 started 10.0.4.203 pmgnyc bionic default Deployed
0/lxd/0 started <redacted-IPv6-prefix>:1000:216:3eff:fe05:8415 juju-31e741-0-lxd-0 bionic default Container started
1 started 10.0.4.26 ruling-molly bionic default Deployed
1/lxd/0 started <redacted-IPv6-prefix>:1000:216:3eff:fe06:c1f8 juju-31e741-1-lxd-0 bionic default Container started
2 started 10.0.4.25 zealot bionic default Deployed
2/lxd/2 started <redacted-IPv6-prefix>:1000:216:3eff:fe8d:d84 juju-31e741-2-lxd-2 bionic default Container started

Offer Application Charm Rev Connected Endpoint Interface Role
prometheus prometheus prometheus2 18 1/1 target http requirer

$ juju remove-application prometheus-sn...

[Achilleas Anagnostopoulos (achilleasa)]

The cause of the issue seems to be the fact that the charm metadata parser used by pre 2.8 juju assumed a default value of 1 for the relation limit when no limit was specified in the metadata (not that 2.7 and earlier did not enforce relation limits).

This has been addressed in 2.8 (it defaults to 0 now if not defined) but if you upgrade a 2.7.x to 2.8 the pre-existing charm metadata in the controller does not get updated so post-upgrade, juju will enforce a limit of 1 to all relations.

A quick fix for now involves doing minor db surgery to reset the stored limit to 0 by running the following script:

db.charms.find().forEach(function (charm){
  ['provides','requires','peers'].forEach(function(relType){
    for (endpoint in charm.meta[relType]){
      // Reset limit; pre 2.8 charm metadata parsers used to set this to 1 by
      // default.
      charm.meta[relType][endpoint].limit=0;
    }
  });

  db.charms.save(charm);
})

[Pen Gale (pengale)]

This is probably worth delaying the 2.8.2 release in order to get in a fix. At least we've found a big gap in testing. :-/ (We need to make sure that our upgrade integration tests attempt to add a new relation to charms that already have a relation -- whoops!)

[Ryan Beisner (1chb1n)]

This change in behavior is potentially service-affecting for production clouds upgrading to Juju 2.8.

[Chris Sanders (chris.sanders)]

I've subscribed field-high due to the impact this has on any 2.8.x upgrades

[Achilleas Anagnostopoulos (achilleasa)]

PR https://github.com/juju/juju/pull/11884 includes a fix as an upgrade step to 2.8.2