disk space for the filesystem mounted on /opt/backups only ~5G (where host-fs reports it as 50 on controller-0)

[Wendy Mitchell]

Brief Description
-----------------.
disk space for the filesystem mounted on /opt/backups appears to be only ~5G (according to df -H)
whereas the system host-fs-list for the backup fs shows it as 50GiB (on controller-0)

Severity
--------
Standard

Steps to Reproduce
------------------
1. clean install of system
2. on controller-0 check the default fs size for backup

eg.
$ system host-fs-list controller-0
+--------------------------------------+---------+-------------+----------------+
| UUID | FS Name | Size in GiB | Logical Volume |
+--------------------------------------+---------+-------------+----------------+
| c4c6fece-b70c-41cc-a91c-d95685b22077 | backup | 50 | backup-lv

3. on the same controller-0 node, run df -H (as root) to confirm the file system mount "Size" /opt/backup

$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 21G 9.1G 11G 47% /
devtmpfs 21G 0 21G 0% /dev
tmpfs 21G 218k 21G 1% /dev/shm
tmpfs 21G 13M 21G 1% /run
tmpfs 21G 0 21G 0% /sys/fs/cgroup
tmpfs 1.1G 119k 1.1G 1% /tmp
/dev/mapper/cgts--vg-ceph--mon--lv 22G 105M 20G 1% /var/lib/ceph/mon
/dev/mapper/cgts--vg-scratch--lv 8.4G 123M 7.8G 2% /scratch
/dev/mapper/cgts--vg-kubelet--lv 11G 39M 9.9G 1% /var/lib/kubelet
/dev/mapper/cgts--vg-backup--lv 5.2G 21M 4.9G 1% /opt/backups
...

Expected Behavior
------------------
Expected output from df -H for backup to roughly correspond with host-fs setting
ie. backup Size from $system host-fs-list controller-0

Actual Behavior
----------------
Discrepancy in size for the backup FS reported in df -H (on controller-0 specifically)

Per discussion with Bob Church
This is the default fs size in filesystem.pp. It must not be setting the hiera data correctly for it.

class platform::filesystem::backup::params (
  $lv_name = 'backup-lv',
  $lv_size = '5',

Reproducibility
---------------
yes

System Configuration
--------------------
This appears on multiple labs (only controller-0 though)
hw: ip-33-36, PV-1 and simplex lab

Branch/Pull Time/Commit
-----------------------
2019-09-05_00-10-00

Last Pass
---------

Timestamp/Logs
--------------

Test Activity
-------------
regression (after new feature changes for Configurable docker_lv and /var/lib/kublet)

[Ghada Khalil (gkhalil)]

Marking as stx.3.0 gating - actual fs size is significantly smaller than reported.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682670

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682670
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=b34c5724179b0416a976a78b16889be659c53bcc
Submitter: Zuul
Branch: master

commit b34c5724179b0416a976a78b16889be659c53bcc
Author: Stefan Dinescu <email address hidden>
Date: Tue Sep 17 18:04:10 2019 +0300

    Resize the filesystem at the same time as the LV

    When resising the logical volume we also need to resize
    the filesystem on top of the volume, otherwise the
    usable space will not actually increase.

    Change-Id: If35ed36204570d36469fbfdcece0d43aaa7d354e
    Closes-Bug: 1843638
    Signed-off-by: Stefan Dinescu <email address hidden>

[Wendy Mitchell (wmitchellwr)]

Retested on 2019-09-22_20-00-00 and still seeing the problem (specifically on controller-0)

$ system host-fs-list controller-0
+--------------------------------------+---------+-------------+----------------+
| UUID | FS Name | Size in GiB | Logical Volume |
+--------------------------------------+---------+-------------+----------------+
| b91ef648-1daa-4ebf-8ba9-eb9de9832857 | backup | 50 | backup-lv

[sysadmin@controller-0 ~(keystone_admin)]$ sudo df -H | grep backup
/dev/mapper/cgts--vg-backup--lv 5.2G 21M 4.9G 1% /opt/backups

[Stefan Dinescu (stefandinescu)]

On what kind of setup was this tested, because I am not seeing this on my end. I tested on vbox on an AIO-SX and on a standard setup (2+2) and everything seemed ok:

On 2+2 setup:
[sysadmin@localhost ~(keystone_admin)]$ system host-fs-list 1
+--------------------------------------+---------+-------------+----------------+
| UUID | FS Name | Size in GiB | Logical Volume |
+--------------------------------------+---------+-------------+----------------+
| b0eb74f7-83cd-4a87-bf9e-6027d6f68aa6 | backup | 50 | backup-lv |
| 918f933f-489f-49c4-9f95-4be3ce67aac5 | docker | 30 | docker-lv |
| 6fce8423-4abe-4b5f-b430-6e4a42efa238 | kubelet | 10 | kubelet-lv |
| a968d647-24d8-410a-a543-2646cfafdc8a | scratch | 8 | scratch-lv |
+--------------------------------------+---------+-------------+----------------+

[sysadmin@localhost ~(keystone_admin)]$ df -h |grep backup
/dev/mapper/cgts--vg-backup--lv 50G 33M 47G 1% /opt/backups

On an AIO-SX:

sysadmin@localhost ~(keystone_admin)]$ system host-fs-list 1
+--------------------------------------+---------+-------------+----------------+
| UUID | FS Name | Size in GiB | Logical Volume |
+--------------------------------------+---------+-------------+----------------+
| 948efd09-e3dc-4bc4-b5e9-a4754ebe12d1 | backup | 50 | backup-lv |
| 8e4ab782-21c7-4f35-86ca-9060c2ec25ea | docker | 30 | docker-lv |
| dabe3163-3a23-4225-9bd3-e212548c792c | kubelet | 10 | kubelet-lv |
| aed712d1-f4fa-4607-a4fd-ca560e64217d | scratch | 8 | scratch-lv |
+--------------------------------------+---------+-------------+----------------+

[sysadmin@localhost ~(keystone_admin)]$ df -h |grep backup
/dev/mapper/cgts--vg-backup--lv 50G 33M 47G 1% /opt/backups

Build-date: BUILD_DATE="2019-09-23 06:33:17 +0000"

Maybe it has something to do with the disk size. Both my setups have a root disk of 300GB. What is the disk-size of the setup you are using?

[Wendy Mitchell (wmitchellwr)]

hw lab: PV-1 2c (platform only install)
2019-09-22_20-00-00

$ system application-list
platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed

$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | compute-1 | worker | unlocked | enabled | available |
| 3 | compute-2 | worker | unlocked | enabled | available |
| 4 | compute-3 | worker | unlocked | enabled | available |
| 5 | compute-4 | worker | unlocked | enabled | available |
| 6 | compute-5 | worker | unlocked | enabled | available |
| 7 | compute-6 | worker | unlocked | enabled | available |
| 8 | compute-7 | worker | unlocked | enabled | available |
| 9 | controller-1 | controller | unlocked | enabled | available |
| 11 | storage-1 | storage | unlocked | enabled | available |
| 12 | compute-0 | worker | unlocked | enabled | available

$ system host-fs-list 1
+--------------------------------------+---------+------+----------------+
| UUID | FS Name | Size | Logical Volume |
| | | in | |
| | | GiB | |
+--------------------------------------+---------+------+----------------+
| b91ef648-1daa-4ebf-8ba9-eb9de9832857 | backup | 50 | backup-lv |
| 35c34bb3-efd9-44b0-baf8-d5362d55afb5 | docker | 30 | docker-lv |
| 41346d40-cbef-430f-84aa-77efa541af9e | kubelet | 10 | kubelet-lv |
| 41a697bd-4cbc-4fb8-a5be-09cfda03ddeb | scratch | 8 | scratch-lv |
+--------------------------------------+---------+------+----------------+

$ sudo df -H | grep backup
Password:
/dev/mapper/cgts--vg-backup--lv 5.2G 21M 4.9G 1% /opt/backups

[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
+--------------------------------------+-----------+---------+---------+-------+------------+------+--------------+--------------------------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
| | de | num | type | gib | gib | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+------+--------------+--------------------------------------------+
| 72cb1ed4-386c-4337-9bf8-a4c0423e023c | /dev/sda | 2048 | HDD | 931. | 931.51 | 7200 | Z1W4GHJC | /dev/disk/by-path/pci-0000:00:1f.2-ata-1.0 |
| | | | | 512 |...

[Stefan Dinescu (stefandinescu)]

Wendy, can you please upload the ansible.log file from this install? I will also do a 2+2+2 install to see if I can reproduce this issue.

[Wendy Mitchell (wmitchellwr)]

verified
2019-09-25_20-00-00
controller-0
df -H
Size Used Avail Use% Mounted on
53G 34M 51G 1% /opt/backups