wrong disk space info

[Andrey Shestakov]

Fuel: 5.0.1
Env: 1 Controller and 4 Compute+Cinder

Deployment of env failed with error
Node 'NODE_controller_0' has insufficient disk space
Volume group 'Base System' requires a minimum of 83968MB
Volume group 'Image Storage' requires a minimum of 5120MB

Steps to reproduce:
1. deploy env
2. reset env few times and deploy again

nailgun log:
2014-08-22 08:34:52.139 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Initialized with node: NODE_controller_0 (id=2, mac=00:25:90:f4:9a:5a)
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Initialized with volumes: [{u'name': u'sda', u'extra': [u'disk/by-id/wwn-0x50014ee2089c89d0', u'disk/by-id/scsi-SATA_WDC_WD1003FBYX-_WD-WCAW36002635', u'disk/by-id/ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW36002635'], u'free_space': 953241, u'volumes': [{u'type': u'boot', u'size': 300}, {u'mount': u'/boot', u'type': u'partition', u'file_system': u'ext2', u'name': u'Boot', u'size': 200}, {u'type': u'lvm_meta_pool', u'size': 0}, {u'vg': u'os', u'type': u'pv', u'lvm_meta_size': 64, u'size': 84032}, {u'vg': u'image', u'type': u'pv', u'lvm_meta_size': 64, u'size': 869337}], u'type': u'disk', u'id': u'sda', u'size': 953869}, {u'_allocate_size': u'min', u'label': u'Base System', u'min_size': 83968, u'volumes': [{u'mount': u'/', u'type': u'lv', u'name': u'root', u'file_system': u'ext4', u'size': 51200}, {u'mount': u'swap', u'type': u'lv', u'name': u'swap', u'file_system': u'swap', u'size': 32768}], u'type': u'vg', u'id': u'os'}, {u'_allocate_size': u'all', u'label': u'Image Storage', u'min_size': 5120, u'volumes': [{u'mount': u'/var/lib/glance', u'type': u'lv', u'name': u'glance', u'file_system': u'xfs', u'size': 869273}], u'type': u'vg', u'id': u'image'}]
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Initialized with disks: [{"name": "sda", "extra": ["disk/by-id/wwn-0x50014ee2089c89d0", "disk/by-id/scsi-SATA_WDC_WD1003FBYX-_WD-WCAW36002635", "disk/by-id/ata-WDC_WD1003FBYX-01Y7B1_WD-WCAW36002635"], "free_space": 953241, "volumes": [{"type": "boot", "size": 300}, {"mount": "/boot", "size": 200, "type": "partition", "file_system": "ext2", "name": "Boot"}, {"type": "lvm_meta_pool", "size": 128}], "type": "disk", "id": "disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0", "size": 953869}]
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_min_os_size with args () returned result: 83968
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_min_os_size with args [] expanded to: 83968
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_swap_size with args () returned result: 32768
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_total_root_vg with args () returned result: 51200
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_total_root_vg with args [] expanded to: 51200
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_swap_size with args () returned result: 32768
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_swap_size with args [] expanded to: 32768
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_min_glance_size with args () returned result: 5120
2014-08-22 08:34:52.140 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_min_glance_size with args [] expanded to: 5120
2014-08-22 08:34:52.141 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_total_vg with args (u'image',) returned result: 869273
2014-08-22 08:34:52.141 DEBUG [7f5c50dfa700] (manager) VolumeManager 140034188766352: Generator calc_total_vg with args [u'image'] expanded to: 869273
2014-08-22 08:34:52.141 DEBUG [7f5c50dfa700] (helpers) Updating task: d75a1f8b-801f-45b6-a3b4-98a7fd011b0a
2014-08-22 08:34:52.142 INFO [7f5c50dfa700] (helpers) Task d75a1f8b-801f-45b6-a3b4-98a7fd011b0a (check_before_deployment) status is set to error
2014-08-22 08:34:52.142 INFO [7f5c50dfa700] (helpers) Task d75a1f8b-801f-45b6-a3b4-98a7fd011b0a (check_before_deployment) progress is set to 100
2014-08-22 08:34:52.143 INFO [7f5c50dfa700] (helpers) Task d75a1f8b-801f-45b6-a3b4-98a7fd011b0a (check_before_deployment) message is set to Node 'NODE_controller_0' has insufficient disk space
Volume group 'Base System' requires a minimum of 83968MB
Volume group 'Image Storage' requires a minimum of 5120MB
2014-08-22 08:34:52.185 DEBUG [7f5c50dfa700] (helpers) Updating cluster status: d75a1f8b-801f-45b6-a3b4-98a7fd011b0a cluster_id: 1 status: error
2014-08-22 08:34:52.190 DEBUG [7f5c50dfa700] (helpers) Updating parent task: cdd146b0-9edc-488a-bd37-ba406403fcfd.
2014-08-22 08:34:52.247 WARNING [7f5c50dfa700] (manager) Checking prerequisites failed: Node 'NODE_controller_0' has insufficient disk space
Volume group 'Base System' requires a minimum of 83968MB
Volume group 'Image Storage' requires a minimum of 5120MB

[Andrey Shestakov (ashestakov)]

fuel-snapshot

[Aleksey Kasatkin (alekseyk-ru)]

Related ticket: https://bugs.launchpad.net/fuel/+bug/1359070

I got the error described here prior to the error in ticket 1359070.
My snapshot in 1359070 is relevant for this issue.

These two may be caused by the same reason.

[Aleksey Kasatkin (alekseyk-ru)]

My env was 5.1 although.

[Mike Scherbakov (mihgen)]

Andrey, thanks for a bug!

Aleksey, all,
how many users does it hit? Is there a simple workaround? As we are approaching release date, would it be possible to move this bug to 6.0, if there is a simple workaround for the issue?

[Aleksey Kasatkin (alekseyk-ru)]

As a workaround: all went ok after remove-and-add-back erroneous node.

[Sebastian Kalinowski (prmtl)]

Disks identifiers has changed:

[{'disk': 'disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0',
  'extra': [...],
  'model': 'WDC WD1003FBYX-0',
  'name': 'sda',
  'removable': '0',
  'size': 1000204886016}]

[{'extra': [..],
  'free_space': 953241,
  'id': 'sda',
  'name': 'sda',
   ...
}]

Will be fixed by the same fix as in https://bugs.launchpad.net/fuel/+bug/1359070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/116809
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=ee906f1fbca842a7933578f547e6f88eadee8c1d
Submitter: Jenkins
Branch: master

commit ee906f1fbca842a7933578f547e6f88eadee8c1d
Author: Sebastian Kalinowski <email address hidden>
Date: Tue Aug 26 09:02:53 2014 +0200

    Force volumes to rebuild after cluster reset

    * rewritten `reset_environment_resp` to use objects
    * after each cluster reset volumes configuration will be rebuild
      to prevent issues when disks identifiers will change
      after reset

    Change-Id: I518a2ad36248b9a676455e8f43ac9f3f76515871
    Closes-Bug: #1359070
    Closes-Bug: #1360248