mon-relation-changed error: cannot read settings for un it "ceph/1" in relation "landscape-client:ceph-client ceph:client": settings not found

[Björn Tillenius]

I got the following trackback using the r92 of lp:~openstack-charmers/charms/trusty/ceph/next:

2015-01-12 10:26:27 INFO mon-relation-changed error: cannot read settings for unit "ceph/1" in relation "landscape-client:ceph-client ceph:client": settings not found
2015-01-12 10:26:27 INFO mon-relation-changed Traceback (most recent call last):
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/mon-relation-changed", line 339, in <module>
2015-01-12 10:26:27 INFO mon-relation-changed hooks.execute(sys.argv)
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/charmhelpers/core/hookenv.py", line 528, in execute
2015-01-12 10:26:27 INFO mon-relation-changed self._hooks[hook_name]()
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/mon-relation-changed", line 209, in mon_relation
2015-01-12 10:26:27 INFO mon-relation-changed notify_client()
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/mon-relation-changed", line 227, in notify_client
2015-01-12 10:26:27 INFO mon-relation-changed client_relation_joined(relid)
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/mon-relation-changed", line 297, in client_relation_joined
2015-01-12 10:26:27 INFO mon-relation-changed client_relation_changed(relid=relid)
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/mon-relation-changed", line 306, in client_relation_changed
2015-01-12 10:26:27 INFO mon-relation-changed settings = relation_get(rid=relid)
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/charmhelpers/core/hookenv.py", line 49, in wrapper
2015-01-12 10:26:27 INFO mon-relation-changed res = func(*args, **kwargs)
2015-01-12 10:26:27 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-0/charm/hooks/charmhelpers/core/hookenv.py", line 314, in relation_get
2015-01-12 10:26:27 INFO mon-relation-changed return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2015-01-12 10:26:27 INFO mon-relation-changed File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2015-01-12 10:26:27 INFO mon-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2015-01-12 10:26:27 INFO mon-relation-changed subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-r', u'client:5', '-']' returned non-zero exit status 1
2015-01-12 10:26:27 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1

The problem seems to be that client_relation_changed() doesn't pass in the unit names to relation_get()

[Björn Tillenius (bjornt)]

landscape:~> juju run --service ceph 'relation-ids client'
- MachineId: "0"
  Stdout: 'client:11

    client:18

    client:5

'
  UnitId: ceph/0
- MachineId: "2"
  Stdout: 'client:11

    client:18

    client:5

'
  UnitId: ceph/1
- MachineId: "3"
  Stdout: 'client:11

    client:18

    client:5

'
  UnitId: ceph/2
landscape:~> juju run --service ceph 'relation-list -r client:5'
- MachineId: "0"
  Stdout: 'landscape-client/1

'
  UnitId: ceph/0
- MachineId: "2"
  Stdout: 'landscape-client/7

'
  UnitId: ceph/1
- MachineId: "3"
  Stdout: 'landscape-client/9

'
  UnitId: ceph/2
landscape:~> juju run --service ceph 'relation-list -r client:18'
- MachineId: "0"
  Stdout: 'cinder/0

'
  UnitId: ceph/0
- MachineId: "2"
  Stdout: 'cinder/0

'
  UnitId: ceph/1
- MachineId: "3"
  Stdout: 'cinder/0

'
  UnitId: ceph/2

[Liam Young (gnuoy)]

Reproduced with this bundle an then adding a ceph unit

openstack-services:
  series: trusty
  services:
    landscape-client:
      charm: landscape-client
    ceph:
      charm: ceph
      num_units: 3
      constraints: mem=1G
      options:
        monitor-count: 3
        fsid: 6547bd3e-1397-11e2-82e5-53567c8d32dc
        monitor-secret: AQCXrnZQwI7KGBAAiPofmKEXKxu5bUzoYLVkbQ==
        osd-devices: /dev/vdb
        osd-reformat: "yes"
        ephemeral-unmount: /mnt
  relations:
      - [ "landscape-client:ceph-client", "ceph:client" ]

juju-deployer -c ceph.yaml
juju add-unit ceph

On the new ceph/3:
root@juju-lytrusty-machine-4:/var/lib/juju/agents/unit-ceph-3/charm# ./hooks/mon-relation-changed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
error: cannot read settings for unit "ceph/1" in relation "landscape-client:ceph-client ceph:client": settings not found
Traceback (most recent call last):
  File "./hooks/mon-relation-changed", line 339, in <module>
    hooks.execute(sys.argv)
  File "/var/lib/juju/agents/unit-ceph-3/charm/hooks/charmhelpers/core/hookenv.py", line 528, in execute
    self._hooks[hook_name]()
  File "./hooks/mon-relation-changed", line 209, in mon_relation
    notify_client()
  File "./hooks/mon-relation-changed", line 227, in notify_client
    client_relation_joined(relid)
  File "./hooks/mon-relation-changed", line 297, in client_relation_joined
    client_relation_changed(relid=relid)
  File "./hooks/mon-relation-changed", line 306, in client_relation_changed
    settings = relation_get(rid=relid)
  File "/var/lib/juju/agents/unit-ceph-3/charm/hooks/charmhelpers/core/hookenv.py", line 49, in wrapper
    res = func(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-3/charm/hooks/charmhelpers/core/hookenv.py", line 314, in relation_get
    return json.loads(subprocess.check_output(_args).decode('UTF-8'))
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-r', u'client:1', '-']' returned non-zero exit status 1