Diving in to this further, I see that the faulty units all sent a request to the Ceph broker /after/ receiving a successful response. glance-api is restarted when a request succeeds.
$ juju run --service glance "grep 'Ceph broker' /var/log/juju/unit-glance*"
- MachineId: 0/lxc/0
Stdout: |
2015-06-02 12:20:03 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:22:58 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:24:10 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
2015-06-02 12:24:59 INFO unit.glance/0.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
UnitId: glance/0
- MachineId: 2/lxc/4
Stdout: |
2015-06-02 12:22:51 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:24:37 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
2015-06-02 12:25:15 INFO unit.glance/1.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
UnitId: glance/1
- MachineId: 3/lxc/6
Stdout: |
2015-06-02 12:20:25 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:22:31 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:24:12 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51)
2015-06-02 12:24:32 INFO unit.glance/2.juju-log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None)
UnitId: glance/2
$ juju run --service glance 'ls /var/lib/glance/images'
- MachineId: 0/lxc/0
Stdout: |
32497170-748d-4a85-ba3d-f1c5f34eb3ed
ccd9d8fe-72e1-4f03-95c5-81629a5e5472
UnitId: glance/0
- MachineId: 2/lxc/4
Stdout: |
e73fa89b-bb54-46a7-b347-b7d04344fe0d
UnitId: glance/1
- MachineId: 3/lxc/6
Stdout: ""
UnitId: glance/2
Diving in to this further, I see that the faulty units all sent a request to the Ceph broker /after/ receiving a successful response. glance-api is restarted when a request succeeds.
$ juju run --service glance "grep 'Ceph broker' /var/log/ juju/unit- glance* " 0.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 0.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 0.juju- log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None) 0.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 1.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 1.juju- log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None) 1.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 2.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 2.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 2.juju- log cmd.go:247 ceph:51: Request(s) sent to Ceph broker (rid=ceph:51) 2.juju- log cmd.go:247 ceph:51: Ceph broker request succeeded (rc=0, msg=None) glance/ images' 748d-4a85- ba3d-f1c5f34eb3 ed 72e1-4f03- 95c5-81629a5e54 72 bb54-46a7- b347-b7d04344fe 0d
- MachineId: 0/lxc/0
Stdout: |
2015-06-02 12:20:03 INFO unit.glance/
2015-06-02 12:22:58 INFO unit.glance/
2015-06-02 12:24:10 INFO unit.glance/
2015-06-02 12:24:59 INFO unit.glance/
UnitId: glance/0
- MachineId: 2/lxc/4
Stdout: |
2015-06-02 12:22:51 INFO unit.glance/
2015-06-02 12:24:37 INFO unit.glance/
2015-06-02 12:25:15 INFO unit.glance/
UnitId: glance/1
- MachineId: 3/lxc/6
Stdout: |
2015-06-02 12:20:25 INFO unit.glance/
2015-06-02 12:22:31 INFO unit.glance/
2015-06-02 12:24:12 INFO unit.glance/
2015-06-02 12:24:32 INFO unit.glance/
UnitId: glance/2
$ juju run --service glance 'ls /var/lib/
- MachineId: 0/lxc/0
Stdout: |
32497170-
ccd9d8fe-
UnitId: glance/0
- MachineId: 2/lxc/4
Stdout: |
e73fa89b-
UnitId: glance/1
- MachineId: 3/lxc/6
Stdout: ""
UnitId: glance/2