Logical volume not extended

[Marian Gasparovic]

When adding second device to block-device parameter, it is added to volume group, but logical volume is not extended to use it

juju ssh cinder/4 "lsblk; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
├─cinder--volumes-cinder--volumes--pool_tmeta 252:0 0 12M 0 lvm
│ └─cinder--volumes-cinder--volumes--pool-tpool 252:2 0 9.5G 0 lvm
│ └─cinder--volumes-cinder--volumes--pool 252:3 0 9.5G 0 lvm
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 9.5G 0 lvm
  └─cinder--volumes-cinder--volumes--pool-tpool 252:2 0 9.5G 0 lvm
    └─cinder--volumes-cinder--volumes--pool 252:3 0 9.5G 0 lvm
sdc 8:32 0 10G 0 disk
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 2 1 0 wz--n- 19.99g 10.47g
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-aotz-- 9.50g 0.00 0.59
Connection to 192.168.100.185 closed.

juju config cinder block-device
sdb sdc

[Marian Gasparovic (marosg)]

Just some additional information - when two disks are provided during deploy, LV is spread across both as expected. It is just adding a block device which does not enlarge LV ^^^

juju ssh cinder/5 "lsblk; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
├─cinder--volumes-cinder--volumes--pool_tmeta 252:0 0 20M 0 lvm
│ └─cinder--volumes-cinder--volumes--pool 252:2 0 19G 0 lvm
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 19G 0 lvm
  └─cinder--volumes-cinder--volumes--pool 252:2 0 19G 0 lvm
sdc 8:32 0 10G 0 disk
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 19G 0 lvm
  └─cinder--volumes-cinder--volumes--pool 252:2 0 19G 0 lvm
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 2 1 0 wz--n- 19.99g 984.00m
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-a-tz-- 18.99g 0.00 0.55

[Liam Young (gnuoy)]

Hi Marian, thanks for reporting the bug. I haven't been able to reproduce this. I have deployed cs:cinder (rev 265) with a single block device and checked the vg:

root@juju-f533ae-1735931-3:~# pvs
  PV VG Fmt Attr PSize PFree
  /dev/vdd cinder-volumes lvm2 a-- 1020.00m 1020.00m
root@juju-f533ae-1735931-3:~# vgs
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 1 0 0 wz--n- 1020.00m 1020.00m

I then added a second device:

juju config cinder block-device="vdd vde"

and the vg was automatically extended:

root@juju-f533ae-1735931-3:~# pvs
  PV VG Fmt Attr PSize PFree
  /dev/vdd cinder-volumes lvm2 a-- 1020.00m 1020.00m
  /dev/vde cinder-volumes lvm2 a-- 1020.00m 1020.00m
root@juju-f533ae-1735931-3:~# vgs
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 2 0 0 wz--n- 1.99g 1.99g

Could you confirm what version if the cinder charm you are using and your version of Ubuntu. If possible could you also attach the contents of /var/log/juju from the cinder unit?

Thanks
Liam

[Liam Young (gnuoy)]

Actually we may be on cross purposes here. I would not expect the logical volumes to be extended as they relate to a specific cinder volume and I see in your paste that you have 2 physical volumes listed in the output from vgs which looks correct.

[Marian Gasparovic (marosg)]

Liam,
I should have said pool, not volume, it is a pool for thin provisioning,
you can see it in my original output. As you can see cinder-volume-pool
is 9.5GB while it should be twice that size. It has 2x size when I put
both block devices to charm from start.

What you showed in you other post, yes, that is also what I see, VG is
extended, that works. But the pool which is then available for thin
volume allocation is not extended.

I looked at charm code, and if I am reading it right, it only
allocates/extends VG, rest is probably executed in Cinder itself later -
that I am just guessing, because I did not find any LV or LV-pool
related code in hooks.
In that case it is not bug in code but maybe missing functionality? I
think common expectation is when you add block device to charm, it
should be reflected in cinder then.

I am 95% sure I used cs:cinder-264, I am on 17.10, juju is 2.3-beta1.
But if what I wrote above is correct, it does not matter, LV/LVpool
extend code is not in cinder charm.

On 05.12.2017 14:57, Liam Young wrote:
> Actually we may be on cross purposes here. I would not expect the
> logical volumes to be extended as they relate to a specific cinder
> volume and I see in your paste that you have 2 physical volumes listed
> in the output from vgs which looks correct.
>
> ** Changed in: charm-cinder
> Status: New => Incomplete
>

[Liam Young (gnuoy)]

Sorry Marian, you are quite right. Ok, I was testing on mitaka and this is evidently a bug on pike+. In pike cinder uses thin provisioning by default with the lvm backend so a lv pool is automatically created that does not correspond to a specific cinder volume. Before any cinder volumes have been provisioned:

ubuntu@gnuoy-bastion:~/bugs/1735931⟫ juju run --unit cinder/0 "pvs"
  PV VG Fmt Attr PSize PFree
  /dev/vdd cinder-volumes lvm2 a-- 3.00g 140.00m

ubuntu@gnuoy-bastion:~/bugs/1735931⟫ juju run --unit cinder/0 "vgs"
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 1 1 0 wz--n- 3.00g 140.00m

ubuntu@gnuoy-bastion:~/bugs/1735931⟫ juju run --unit cinder/0 "lvs"
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-a-tz-- 2.85g 0.00 1.07

Creating a cinder volume creates a lv inside the logical thin pool:

ubuntu@gnuoy-bastion:~/bugs/1735931⟫ cinder create 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-12-05T17:35:02.000000 |
| description | None |
| encrypted | False |
| id | 0f2c6b0a-09c7-4bb8-b5a8-718d7a4cfd4f |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 5087487c5630440e955f5b33d445fe9f |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 89ea60545efd4752b4cc90329958b698 |
| volume_type | None |
+--------------------------------+--------------------------------------+

ubuntu@gnuoy-bastion:~/bugs/1735931⟫ juju run --unit cinder/0 "lvs"...

[Liam Young (gnuoy)]

I have pushed a fix up to the charmstore in my namespace: cs:~gnuoy/cinder-2
If anyone has time to give a try that would be great.

[Marian Gasparovic (marosg)]

Liam, this is what I see, it removed thin provisioning, was that the plan?

juju deploy cs:~gnuoy/cinder-2
Located charm "cs:~gnuoy/cinder-2".
Deploying charm "cs:~gnuoy/cinder-2".

juju ssh cinder/0 "lsblk; sudo pvs; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
sdc 8:32 0 10G 0 disk

juju config cinder overwrite=true

juju ssh cinder/0 "lsblk; sudo pvs; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
sdc 8:32 0 10G 0 disk
  PV VG Fmt Attr PSize PFree
  /dev/sdb cinder-volumes lvm2 a-- 10.00g 10.00g
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 1 0 0 wz--n- 10.00g 10.00g

juju config cinder block-device="sdb sdc"

juju ssh cinder/0 "lsblk; sudo pvs; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
sdc 8:32 0 10G 0 disk
  PV VG Fmt Attr PSize PFree
  /dev/sdb cinder-volumes lvm2 a-- 10.00g 10.00g
  /dev/sdc cinder-volumes lvm2 a-- 10.00g 10.00g
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 2 0 0 wz--n- 19.99g 19.99g

[Liam Young (gnuoy)]

Are you definitely deploying pike or later in your last test?

[Marian Gasparovic (marosg)]

Of course I forgot to specify openstack-origin, sorry about that. I will
do the test again.

On 07.12.2017 17:29, Liam Young wrote:
> Are you definitely deploying pike or later in your last test?
>
> ** Changed in: charm-cinder
> Importance: Undecided => High
>
> ** Changed in: charm-cinder
> Assignee: (unassigned) => Liam Young (gnuoy)
>
> ** Changed in: charm-cinder
> Milestone: None => 18.05
>

[Marian Gasparovic (marosg)]

Looks good!

juju config cinder block-device
sdb

juju ssh cinder/0 "lsblk; sudo pvs; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
├─cinder--volumes-cinder--volumes--pool_tmeta 252:0 0 12M 0 lvm
│ └─cinder--volumes-cinder--volumes--pool 252:2 0 9.5G 0 lvm
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 9.5G 0 lvm
  └─cinder--volumes-cinder--volumes--pool 252:2 0 9.5G 0 lvm
sdc 8:32 0 10G 0 disk
  PV VG Fmt Attr PSize PFree
  /dev/sdb cinder-volumes lvm2 a-- 10.00g 484.00m
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 1 1 0 wz--n- 10.00g 484.00m
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-a-tz-- 9.50g 0.00 0.59

juju config cinder block-device="sdb sdc"

juju ssh cinder/0 "lsblk; sudo pvs; sudo vgs; sudo lvs"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
sdb 8:16 0 10G 0 disk
├─cinder--volumes-cinder--volumes--pool_tmeta 252:0 0 12M 0 lvm
│ └─cinder--volumes-cinder--volumes--pool 252:2 0 19.5G 0 lvm
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 19.5G 0 lvm
  └─cinder--volumes-cinder--volumes--pool 252:2 0 19.5G 0 lvm
sdc 8:32 0 10G 0 disk
└─cinder--volumes-cinder--volumes--pool_tdata 252:1 0 19.5G 0 lvm
  └─cinder--volumes-cinder--volumes--pool 252:2 0 19.5G 0 lvm
  PV VG Fmt Attr PSize PFree
  /dev/sdb cinder-volumes lvm2 a-- 10.00g 484.00m
  /dev/sdc cinder-volumes lvm2 a-- 10.00g 0
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 2 1 0 wz--n- 19.99g 484.00m
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-a-tz-- 19.50g 0.00 0.91

[Edward Hope-Morley (hopem)]

Patch: https://review.openstack.org/#/c/526448

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526448
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=f9937c90079ed60237b10e6c39e19cc4f6d8f938
Submitter: Zuul
Branch: master

commit f9937c90079ed60237b10e6c39e19cc4f6d8f938
Author: Liam Young <email address hidden>
Date: Thu Dec 7 16:25:38 2017 +0000

    Extend the cinder lvm thin pool with new devices

    If a new device is added to the block-device configuration option
    then add the new device to the cinder thin pool if it exists.

    From pike+ if the lvm backend is being used then it defaults to
    using a thin provisioning pool. This patch makes sure any new
    devices are used to extend that pool.

    Change-Id: I851e4eb58dbdc122086d53fe346ea045c6044f50
    Partial-Bug: #1735931