Fuel for OpenStack

  1. Bug #1430845
  2. Comment #6

Comment 6 for bug 1430845

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Does it works properly in you environment?

I tried to test more complicated case, it looks that images storage did not survived interruption of re-balancing.

Steps:
1. Created Environment:
Ubuntu 14.04.1, Neutron VLAN
1 Controller + Ceph OSD
1 Compute + Ceph OSD
Replication Factor = 2
3 nodes left not allocated.
All this were launched on Virtualbox on my desktop.

2. Filled Glance by Images (they took 70% of Ceph storage)
# df -h
...
/dev/sdc3 64G 44G 21G 69% /var/lib/ceph/osd/ceph-0
/dev/sdb3 64G 45G 19G 71% /var/lib/ceph/osd/ceph-1

Also calculated and saved md5 sums of uploaded images.

3. Then i added 3 new OSD nodes.
4. Deployment of additional nodes passed OK.
5. After new OSD nodes were deployed - Ceph rebalancing started. It heavily loaded hardware resources host machine.
6. For test after some time of re-balancing i have switched-off additional OSD nodes.
7. Check Ceph status
# ceph -s
    cluster 1f5161b7-a500-4757-a486-0b6d44430a15
     health HEALTH_WARN 481 pgs degraded; 25 pgs down; 25 pgs peering; 734 pgs stale; 25 pgs stuck inactive; 734 pgs stuck stale; 506 pgs stuck unclean; recovery 1525/20738 objects degraded (7.354%)
     monmap e1: 1 mons at {node-1=192.168.0.3:6789/0}, election epoch 1, quorum 0 node-1
     osdmap e131: 10 osds: 4 up, 4 in
      pgmap v5033: 2496 pgs, 12 pools, 82212 MB data, 10369 objects
            133 GB used, 119 GB / 253 GB avail
            1525/20738 objects degraded (7.354%)
                 253 stale+active+clean
                1737 active+clean
                  25 down+peering
                 481 stale+active+degraded

8. List images - ok.
Try to download image:
# glance image-download --file ~/dummy${i}.iso dummy${i}.iso
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
 (HTTP N/A)
Try to list images one more time:
# glance image-list
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
 (HTTP 503)
Horizon also refuses to retrieve images list.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "461"
  build_id: "2015-05-24_10-19-44"
  nailgun_sha: "76441596e4fe6420cc7819427662fa244e150177"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "0bd72c72369e743376864e8e8dabfe873d40450a"
  fuel-library_sha: "889c2534ceadf8afd5d1540c1cadbd913c0c8c14"
  fuel-ostf_sha: "9a5f55602c260d6c840c8333d8f32ec8cfa65c1f"
  fuelmain_sha: "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"