[volume manager] doesn't take into account opt io sizes for block devices. But LVM does.

[Alexander Gordeev]

fuel-7.0

nailgun generated too tight partitioning scheme, therefore fuel-agent failed to create lv for `os` vg.

2016-02-15 10:02:38 INFO fuel_agent.cmd.agent
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent NotEnoughSpaceError: Error while creating lv: vg os has only 16368 m of free space, but at least 16384 m is needed
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent 'but at least %s m is needed' % (vgname, vg[0]['free'], size))
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent File "/usr/lib/python2.6/site-packages/fuel_agent/utils/lvm.py", line 209, in lvcreate
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent lu.lvcreate(lv.vgname, lv.name, lv.size)
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent File "/usr/lib/python2.6/site-packages/fuel_agent/manager.py", line 243, in do_partitioning
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent self.do_partitioning()
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent File "/usr/lib/python2.6/site-packages/fuel_agent/manager.py", line 510, in do_provisioning
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent getattr(mgr, action)()
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent File "/usr/lib/python2.6/site-packages/fuel_agent/cmd/agent.py", line 97, in main
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent Traceback (most recent call last):
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent [-] Error while creating lv: vg os has only 16368 m of free space, but at least 16384 m is needed

ks spaces (partitioning scheme generated by nailgun for a particular node)
[
  {
    "name": "sdb",
    "extra": [
      "disk/by-id/wwn-0x60002ac0000000000000002b0000567f",
      "disk/by-id/scsi-360002ac0000000000000002b0000567f"
    ],
    "free_space": 0,
    "volumes": [
      {
        "type": "boot",
        "size": 300
      },
      {
        "mount": "/boot",
        "type": "partition",
        "file_system": "ext2",
        "name": "Boot",
        "size": 200
      },
      {
        "type": "lvm_meta_pool",
        "size": 0
      },
      {
        "vg": "os",
        "type": "pv",
        "lvm_meta_size": 64,
        "size": 69819
      },
      {
        "vg": "cinder",
        "type": "pv",
        "lvm_meta_size": 64,
        "size": 649587
      },
      {
        "vg": "vm",
        "type": "pv",
        "lvm_meta_size": 64,
        "size": 328670
      }
    ],
    "type": "disk",
    "id": "disk/by-path/pci-0000:08:00.0-fc-0x20110002ac00567f-lun-0",
    "size": 1048576
  },
  {
    "_allocate_size": "min",
    "label": "Base System",
    "min_size": 67584,
    "volumes": [
      {
        "mount": "/",
        "type": "lv",
        "name": "root",
        "file_system": "ext4",
        "size": 53371
      },
      {
        "mount": "swap",
        "type": "lv",
        "name": "swap",
        "file_system": "swap",
        "size": 16384
      }
    ],
    "type": "vg",
    "id": "os"
  },
  {
    "_allocate_size": "all",
    "label": "Cinder",
    "min_size": 1536,
    "volumes": [],
    "type": "vg",
    "id": "cinder"
  },
  {
    "_allocate_size": "all",
    "label": "Virtual Storage",
    "min_size": 5120,
    "volumes": [
      {
        "mount": "/var/lib/nova",
        "type": "lv",
        "name": "nova",
        "file_system": "xfs",
        "size": 328606
      }
    ],
    "type": "vg",
    "id": "vm"
  }
]

[Alexander Gordeev (a-gordeev)]

size of /dev/sdb4 was 69819M

4:425MiB:70244MiB:69819MiB::primary:;

pv was created with --metadatacopies 2 --metadatasize 28m (so, 28*2 expected to be reserved for lvm service needs)

2016-02-15 10:02:37 INFO fuel_agent.utils.lvm [-] Found physical volumes: [{'uuid': 'D9yvra-Y48g-PZe1-BnFq-ogah-cCeZ-jIAAOO', 'devsize': 69819, 'name': '/dev/sdb4', 'psize': 69819, 'vg': None}]
2016-02-15 10:02:37 INFO fuel_agent.utils.utils [-] Trying to execute command: pvdisplay -C --noheading --units m --options pv_name,vg_name,pv_size,dev_size,pv_uuid --separator ;
2016-02-15 10:02:37 INFO fuel_agent.utils.utils [-] Trying to execute command: pvcreate --metadatacopies 2 --metadatasize 28m /dev/sdb4

but then

2016-02-15 10:02:38 INFO fuel_agent.utils.lvm [-] Found volume groups: [{'size': 69740, 'name': 'os', 'free': 69740, 'uuid': 'HHJqnA-tEDc-ixFO-U7M0-FHYM-ruLS-EtQeyo'}]
2016-02-15 10:02:38 INFO fuel_agent.utils.utils [-] Trying to execute command: vgdisplay -C --noheading --units m --options vg_name,vg_uuid,vg_size,vg_free --separator ;
2016-02-15 10:02:38 INFO fuel_agent.utils.utils [-] Trying to execute command: vgcreate os /dev/sdb4

vg os has got only 69740M (expected value was 69819 - 56 = 69763, that is 23M shorter than expected)

therefore, fuel-agent ended up with
2016-02-15 10:02:38 INFO fuel_agent.cmd.agent NotEnoughSpaceError: Error while creating lv: vg os has only 16368 m of free space, but at least 16384 m is needed

RC: for some reasons, created lvm logical volume occupied more space than expected. Underestimated size is roughly within 12-16M.

[Alexander Gordeev (a-gordeev)]

tried changing lvm_meta_size to 128

https://github.com/openstack/fuel-web/blob/stable/7.0/nailgun/nailgun/extensions/volume_manager/manager.py#L816

failed once again, with

NotEnoughSpaceError: Error while creating lv: vg os has only 16380 m of free space, but at least 16384 m is needed

http://paste.openstack.org/show/487091/

pvcreate --metadatacopies 2 --metadatasize 60m /dev/sda4

4:425MiB:68137MiB:67712MiB::primary:;

2016-02-16 11:43:42 INFO fuel_agent.utils.lvm [-] Found volume groups: [{'size': 67580, 'name': 'os', 'free': 67580, 'uuid': 'GkoCA6-KuUB-Ze07-HMl5-li2T-AOBs-Zy6Ufr'}]
2016-02-16 11:43:42 INFO fuel_agent.utils.utils [-] Trying to execute command: vgdisplay -C --noheading --units m --options vg_name,vg_uuid,vg_size,vg_free --separator ;
2016-02-16 11:43:42 INFO fuel_agent.utils.utils [-] Trying to execute command: vgcreate os /dev/sda4

[14:48:45] Alexander Gordeev: In [19]: 67580-67712
Out[19]: -132

132, but metadata took only 120. The same 12-16M.

Could it be related to optimal i/o size alignment?

lsblk -t reported 16M opt i/o size
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE
sda 0 16384 16777216 512 512 1 cfq 128 128
├─sda1 0 16384 16777216 512 512 1 cfq 128 128
├─sda2 15872 16384 16777216 512 512 1 cfq 128 128
├─sda3 15872 16384 16777216 512 512 1 cfq 128 128
├─sda4 15872 16384 16777216 512 512 1 cfq 128 128
│ └─os-root (dm-0) 0 16384 16777216 512 512 1 128 128
├─sda5 15872 16384 16777216 512 512 1 cfq 128 128
├─sda6 15872 16384 16777216 512 512 1 cfq 128 128
└─sda7 15872 16384 16777216 512 512 1 cfq 128 128

[Alexander Gordeev (a-gordeev)]

yep, yep, yep. Sounds like yet another alignment issue. (previous was caught on partition level. https://bugs.launchpad.net/fuel/+bug/1543233 )

```

Logical Volume Manager (LVM)
============================
LVM provides userspace tools that are used to manage the kernel's DM
devices. LVM will shift the start of the data area, that a given DM
device will use, to account for a non-zero 'alignment_offset' associated
with any device LVM manages. This means LVM logical volumes will be
properly aligned (alignment_offset=0). LVM will adjust for any
'alignment_offset' by default but this may be disabled through
lvm.conf's 'data_alignment_offset_detection'. Disabling this is not
recommended.

LVM will also detect the "I/O hints" for a device. The start of a
device's data area will be a multiple of the 'minimum_io_size' or
'optimal_io_size' exposed in sysfs. 'minimum_io_size' is used if
'optimal_io_size' is undefined (0). LVM will automatically determine
these "I/O hints" by default but this may be disabled through lvm.conf's
'data_alignment_detection'. Disabling this is not recommended.

``` http://people.redhat.com/msnitzer/docs/io-limits.txt

It looks like nailgun (or at least volume manager) should take into account min io/opt io values exposed in sysfs for every block device and use them for doing more precise volume size calculations.

otherwise, we'll end up with the wierd issues like those 2: this one and https://bugs.launchpad.net/fuel/+bug/1543233 Both observed only when those optimal io values become higher than 1M.

[Alexander Gordeev (a-gordeev)]

probably https://bugs.launchpad.net/fuel/+bug/1510933 is a kind of duplicate of this one.

[Alexander Gordeev (a-gordeev)]

obviously, it's not a medium. High or even critical. Major feature is broken, workaround is not found yet, affects only specific h/w

in that particular case, block device is connected through FC

[Alexander Gordeev (a-gordeev)]

Yury Konov told me that it was customer-found issue. Hence tags were adjusted.

[Alexander Gordeev (a-gordeev)]

as for work-around: one could try to reduce size of any logical volume for os volume group by 100-200M directly inside nailgun db for every node in the cluster. Perhaps, that needs to be applied to any other volume groups (eg: logs, cinder, etc).

unfortunatelly, that couldn't be achieved by editing disks.yaml files for particular node as those files represent only high level volume roles and don't allow to change logical volume sizes of which this volume role consists of.

[Dmitry Guryanov (dguryanov)]

Why can't we get volume group size from lvm (by vgdisplay command)? I think trying to guess it or duplicate lvm logic is not a good idea.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286139

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286139
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=d1d17f253c8c837da10cc50239dca5e2ac9af095
Submitter: Jenkins
Branch: master

commit d1d17f253c8c837da10cc50239dca5e2ac9af095
Author: Dmitry Guryanov <email address hidden>
Date: Mon Mar 28 20:15:56 2016 +0300

    report optimal io size for block devices

    LVM aligns devices data area to io hints, reported for
    block devices. Optimal io size can be big for some devices
    (16M, as reported in bug). So nailgun's volume manager
    should take it into account while doing partitioning.
    So nailgun-agent should send this information to nailgun.

    Change-Id: Idd9b778f7f21d4fe9d3fd029038664de41f93447
    Partial-Bug: #1546049

[Dmitry Pyzhov (dpyzhov)]

Requires changes in interfaces and proper design. Moving to Newtown release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/380450

[Alexander Gordeev (a-gordeev)]

For some cases, it's better to completely turn off all of these lvm alignments.

We should provide a working step-by-step example of how to build new bootstrap image with custom lvm config.

[Alexander Gordeev (a-gordeev)]

Instructions how-to build new bootstrap image with customized LVM config in order to ignore i/o hints.

1) download lvm2 package for trusty to obtain default lvm.conf

http://packages.ubuntu.com/trusty/lvm2

2) extract etc/lvm/lvm.conf, open it, find and change

  data_alignment_detection = 1

to

  data_alignment_detection = 0

3) recreate directory structure for lvm confg
$ mkdir -p override_lvm/etc/lvm
$ cp <updated lvm.conf> override_lvm/etc/lvm
4) rebuild and active new bootstrap image:

$ fuel-bootstrap --verbose --debug build --extra-dir override_lvm/ --label lvm_without_data_alignment --activate

5) remove and add nodes, so they will be loaded into that updated bootstrap image.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/380450