gate-tempest-dsvm-full-ceph fails with Insufficient free space for volume creation on host ubuntu@ceph#ceph (requested / avail): 1/0.0

[Matt Riedemann]

Seeing this in the gate:

http://logs.openstack.org/80/193380/8/check/gate-tempest-dsvm-full-ceph/787f33b/logs/screen-c-sch.txt.gz#_2015-07-15_23_29_43_004

Volume create fails during scheduling because there are no available hosts:

2015-07-15 23:29:43.004 WARNING cinder.scheduler.filters.capacity_filter [req-9d90ac1a-33ed-4476-8322-44a08451e562 tempest-VolumesV2ListTestJSON-842067082] Insufficient free space for volume creation on host ubuntu@ceph#ceph (requested / avail): 1/0.0
2015-07-15 23:29:43.005 WARNING cinder.scheduler.filter_scheduler [req-9d90ac1a-33ed-4476-8322-44a08451e562 tempest-VolumesV2ListTestJSON-842067082] No weighed hosts found for volume with properties: {u'name': u'ceph', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2015-07-15T23:15:34.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'ceph'}, u'is_public': True, u'deleted_at': None, u'id': u'802fb4fa-7264-4703-afab-c37bedaf2841', u'description': None}
2015-07-15 23:29:43.009 ERROR cinder.scheduler.flows.create_volume [req-9d90ac1a-33ed-4476-8322-44a08451e562 tempest-VolumesV2ListTestJSON-842067082] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid host was found. No weighed hosts available

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIHJ1biB0YXNrIGNpbmRlci5zY2hlZHVsZXIuZmxvd3MuY3JlYXRlX3ZvbHVtZS5TY2hlZHVsZUNyZWF0ZVZvbHVtZVRhc2s7dm9sdW1lOmNyZWF0ZTogTm8gdmFsaWQgaG9zdCB3YXMgZm91bmQuIE5vIHdlaWdoZWQgaG9zdHMgYXZhaWxhYmxlXCIgQU5EIHRhZ3M6XCJzY3JlZW4tYy1zY2gudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzcwNTY0NjAyMTJ9

1856 hits in 7 days, check and gate, all failures, master and stable/kilo branches.

[Matt Riedemann (mriedem)]

This is a better query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW5zdWZmaWNpZW50IGZyZWUgc3BhY2UgZm9yIHZvbHVtZSBjcmVhdGlvbiBvbiBob3N0XCIgQU5EIG1lc3NhZ2U6XCIocmVxdWVzdGVkIC8gYXZhaWwpOiAxLzAuMFwiIEFORCB0YWdzOlwic2NyZWVuLWMtc2NoLnR4dFwiIEFORCBOT1QgYnVpbGRfcXVldWU6XCJwZXJpb2RpYy1zdGFibGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNzA2NzAzMjU4N30=

This really only hits on the ceph job.

[Ivan Kolodyazhny (e0ne)]

It's a ceph issue:
 http://tracker.ceph.com/issues/10257

It is fixed in 0.80.10

[Ivan Kolodyazhny (e0ne)]

Upgrade to 0.80.10 didn't help. Change back to Confirmed

[Ivan Kolodyazhny (e0ne)]

http://paste.openstack.org/show/475077/

[Ivan Kolodyazhny (e0ne)]

command "ceph osd crush reweight osd.0 1" fixed issue

[Ivan Kolodyazhny (e0ne)]

It's probably ceph or devstack issue

[Jon Bernard (jbernard)]

This looks like a devstack deploy issue, the osd is up and in, but its weight is set to 0, which is why MAX_AVAIL is 0 for all pools. The command listed above should resolve the issue.

[Matt Riedemann (mriedem)]

I'm not sure why this was marked Incomplete. This is still one of the top failures in the check/gate queues.

[Jon Bernard (jbernard)]

https://review.openstack.org/#/c/240328/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jon Bernard (<email address hidden>) on branch: master
Review: https://review.openstack.org/240328

[Markus Zoeller (markus_z) (mzoeller)]

This is still happening with around 19hits/10days: http://status.openstack.org/elastic-recheck/#1475333

[Matt Riedemann (mriedem)]

Not seeing this anymore so marking it invalid now.