[2.7.3] relation-get --app fails on non-leader units (peer relations)

[Dmitrii Shcherbakov]

See the attachment which contains a screencast.

On a non-leader unit:

* peer-relation-changed fires;
* `relation-get --app=True` is called and returns 0;
   * the same happens for a more explicit variation `relation-get -r 0 - ceph-iscsi --app=True --format=json`
* charm exits with 0 exit code;
* unit agent fails the peer-relation-changed hook execution with the following errors:

unit-ceph-iscsi-0: 23:49:22 ERROR juju.worker.uniter.context could not write settings from "cluster-relation-changed" to relation 0: permission denied
unit-ceph-iscsi-0: 23:49:22 ERROR juju.worker.uniter.operation hook "cluster-relation-changed" failed: could not write settings from "cluster-relation-changed" to relation 0: permission denied

JUJU_REMOTE_APP is set appropriately:

env | grep JUJU_
JUJU_METER_INFO=not set
JUJU_CHARM_HTTPS_PROXY=
JUJU_CONTEXT_ID=ceph-iscsi/0-cluster-relation-changed-2572787010059366350
JUJU_MODEL_UUID=c2189b8c-392e-422f-8054-a58842cb36fa
JUJU_VERSION=2.7.3
JUJU_CHARM_HTTP_PROXY=
JUJU_UNIT_NAME=ceph-iscsi/0
JUJU_RELATION_ID=cluster:0
JUJU_HOOK_NAME=cluster-relation-changed
JUJU_AGENT_SOCKET_ADDRESS=@/var/lib/juju/agents/unit-ceph-iscsi-0/agent.socket
JUJU_RELATION=cluster
JUJU_CHARM_FTP_PROXY=
JUJU_AVAILABILITY_ZONE=
JUJU_METER_STATUS=AMBER
JUJU_CHARM_DIR=/var/lib/juju/agents/unit-ceph-iscsi-0/charm
JUJU_DEBUG=/tmp/juju-debug-hooks-443265531
JUJU_API_ADDRESSES=10.209.240.35:17070
JUJU_REMOTE_UNIT=
JUJU_PRINCIPAL_UNIT=
JUJU_REMOTE_APP=ceph-iscsi
JUJU_MODEL_NAME=iscsi
JUJU_MACHINE_ID=0
JUJU_SLA=unsupported
JUJU_AGENT_SOCKET_NETWORK=unix
JUJU_CHARM_NO_PROXY=127.0.0.1,localhost,::1

The expected behavior is that non-leader units should be able to read peer app relation data for their own application.

Note: non-leader units should not be able to read app relation data for *their own app* for non-peer relations. I think this restriction is just applied to peer relations and causes this issue because units do have to read their own app's relation data which is a special case.

[Dmitrii Shcherbakov (dmitriis)]

Looks like this is a regression compared to 2.7.2.

I used not to get this error with a charm I was working on. This could be used as a reproducer:

git clone https://github.com/dshcherb/charm-cockroachdb
git submodule update --init
juju deploy ./ --config default_zone_replicas=3 --config system_data_replicas=3 -n 3

[Achilleas Anagnostopoulos (achilleasa)]

This regression may have been introduced by the fix for https://bugs.launchpad.net/juju/+bug/1854348. I will investigate further next week.

[John A Meinel (jameinel)]

It sounds like the culprit is this:
func (ctx *ContextRelation) WriteSettings() error {
 var appSettings params.Settings
 if ctx.applicationSettings != nil {
  appSettings = ctx.applicationSettings.FinalResult()
 }
 var unitSettings params.Settings
 if ctx.settings != nil {
  unitSettings = ctx.settings.FinalResult()
 }
 return errors.Trace(ctx.ru.UpdateRelationSettings(unitSettings, appSettings))
}

Where if you have called "relation-get --app" then context.applicationSettings ends up non-nil. Which then tries to call UpdateRelationSettings (even though there aren't any actual changes to the settings.)
Maybe FinalResult isn't properly recognizing that there aren't any changes to the settings?

[John A Meinel (jameinel)]

(looking at FinalResult it always just copies the whole map, it doesn't make any check against whether there are changes or not. So a leader calling "relation-get --app" actually does a WriteSettings at the end, whether or not it called relation-set.)

[Dmitrii Shcherbakov (dmitriis)]

https://github.com/dshcherb/juju/commit/af1be8f26d3d6c396b50562e274e17e2dd02275d - a 2.7 branch. Manually tested but needs some unit tests.

[Achilleas Anagnostopoulos (achilleasa)]

PR https://github.com/juju/juju/pull/11289 includes a fix for the 2.7 branch

[Achilleas Anagnostopoulos (achilleasa)]

PR https://github.com/juju/juju/pull/11292 forward ports the fix to develop.

[Stuart Bishop (stub)]

I'm seeing this behavior in non-peer relations too, where non-leaders need to inspect their own application relation data

[Stuart Bishop (stub)]

Opened Bug #1869915, as I was able to reproduce with non-peer relations with Juju 2.7.5+2.7-3a6aa80