Comment 7 for bug 1456379

Revision history for this message
Adam Collard (adam-collard) wrote :

Diving in to this further, I see that the faulty units all sent a request to the Ceph broker /after/ receiving a successful response. glance-api is restarted when a request succeeds.

$ juju run --service glance "grep 'Ceph broker' /var/log/juju/unit-glance*"
- MachineId: 0/lxc/0
  Stdout: |
    2015-06-02 12:20:03 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:22:58 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:24:10 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
    2015-06-02 12:24:59 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
  UnitId: glance/0
- MachineId: 2/lxc/4
  Stdout: |
    2015-06-02 12:22:51 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:24:37 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
    2015-06-02 12:25:15 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
  UnitId: glance/1
- MachineId: 3/lxc/6
  Stdout: |
    2015-06-02 12:20:25 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:22:31 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:24:12 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
    2015-06-02 12:24:32 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
  UnitId: glance/2
$ juju run --service glance 'ls /var/lib/glance/images'
- MachineId: 0/lxc/0
  Stdout: |
    32497170-748d-4a85-ba3d-f1c5f34eb3ed
    ccd9d8fe-72e1-4f03-95c5-81629a5e5472
  UnitId: glance/0
- MachineId: 2/lxc/4
  Stdout: |
    e73fa89b-bb54-46a7-b347-b7d04344fe0d
  UnitId: glance/1
- MachineId: 3/lxc/6
  Stdout: ""
  UnitId: glance/2