StarlingX

  1. Bug #1926591
  2. Activity log

Activity log for bug #1926591

Date Who What changed Old value New value Message
2021-04-29 10:26:53 Mihnea Saracin bug added bug
2021-04-29 10:26:57 Mihnea Saracin starlingx: assignee Mihnea Saracin (msaracin)
2021-04-29 13:56:45 OpenStack Infra starlingx: status New In Progress
2021-04-29 15:34:07 Ghada Khalil tags stx.update
2021-04-29 15:34:50 Ghada Khalil description Brief Description ----------------- After restore playbook was executed, the unlock failed because docker-lv resize was not successful. Severity -------- Critical Steps to Reproduce ------------------ Install a system Backup it Run restore playbook Unlock controller Expected Behavior ------------------ Unlock succeeds Actual Behavior ---------------- Unlock fails Reproducibility --------------- 100% System Configuration -------------------- AIO-SX, AIO-DX Branch/Pull Time/Commit ----------------------- stx master build on "2021-03-21" Last Pass --------- 2021-04-15 worked, did not test between 15-20 Timestamp/Logs -------------- Puppet wanted to resize docker-lv from 20G to 30G and failed at unlock: 2021-04-22T04:59:09.337 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): Current: value=20.0, unit=G, kibi=20971520 2021-04-22T04:59:09.342 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): New: value=30.0, unit=G, kibi=31457280 2021-04-22T04:59:09.345 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.363 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 31457280k /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.417 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.425 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y check /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.471 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/docker-lv 31457280k' 2021-04-22T04:59:09.507 Notice: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]/size: size changed '20G' to '30G' 2021-04-22T04:59:09.512 Debug: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]: The container Platform::Filesystem[docker-lv] will propagate my refresh event 2021-04-22T04:59:09.516 Debug: 2021-04-22 04:59:09 +0000 Class[Platform::Lvm::Vg::Cgts_vg]: The container Stage[main] will propagate my refresh event 2021-04-22T04:59:09.520 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:09.536 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/etcd-lv' 2021-04-22T04:59:09.560 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:09.584 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.611 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.632 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): Current: value=2.0, unit=G, kibi=2097152 2021-04-22T04:59:09.638 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): New: value=10.0, unit=G, kibi=10485760 2021-04-22T04:59:09.642 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.653 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 10485760k /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.710 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:10.060 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y check /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:10.166 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/kubelet-lv 10485760k' 2021-04-22T04:59:10.355 Notice: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]/size: size changed '2G' to '10G' 2021-04-22T04:59:10.359 Debug: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]: The container Platform::Filesystem[kubelet-lv] will propagate my refresh event 2021-04-22T04:59:10.364 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.367 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.371 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.376 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.379 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped kubelet-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.382 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.385 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.389 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.394 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:10.415 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/extension-lv' 2021-04-22T04:59:10.441 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.445 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.457 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped docker-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.466 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.470 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/blkid /dev/cgts-vg/docker-lv' 2021-04-22T04:59:10.476 Debug: 2021-04-22 04:59:10 +0000 Executing: 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1' 2021-04-22T04:59:10.485 Error: 2021-04-22 04:59:10 +0000 Execution of 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1' returned 1: mkfs.xfs: /dev/cgts-vg/docker-lv contains a mounted filesystem 2021-04-22T04:59:10.489 Usage: mkfs.xfs 2021-04-22T04:59:10.492 /* blocksize */ [-b log=n|size=num] Firstly, the resize should not be triggered here, in an older load (2021-04-15 )the docker-lv was 30G before unlock. For some reason docker-lv fs type is drbd and I think it's wrong, it should be xfs: controller-0:/var/log# blkid /dev/cgts-vg/docker-lv /dev/cgts-vg/docker-lv: UUID="c5c72dc8a5af335b" TYPE="drbd" Possible similar bug: https://bugs.launchpad.net/starlingx/+bug/1883825 Test Activity ------------- Developer Testing Brief Description ----------------- After restore playbook was executed, the unlock failed because docker-lv resize was not successful. Severity -------- Critical Steps to Reproduce ------------------ Install a system Backup it Run restore playbook Unlock controller Expected Behavior ------------------ Unlock succeeds Actual Behavior ---------------- Unlock fails Reproducibility --------------- 100% System Configuration -------------------- AIO-SX, AIO-DX Branch/Pull Time/Commit ----------------------- stx master build on "2021-04-21" Last Pass --------- 2021-04-15 worked, did not test between 15-20 Timestamp/Logs -------------- Puppet wanted to resize docker-lv from 20G to 30G and failed at unlock: 2021-04-22T04:59:09.337 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): Current: value=20.0, unit=G, kibi=20971520 2021-04-22T04:59:09.342 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): New: value=30.0, unit=G, kibi=31457280 2021-04-22T04:59:09.345 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.363 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 31457280k /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.417 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.425 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y check /dev/cgts-vg/docker-lv' 2021-04-22T04:59:09.471 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/docker-lv 31457280k' 2021-04-22T04:59:09.507 Notice: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]/size: size changed '20G' to '30G' 2021-04-22T04:59:09.512 Debug: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]: The container Platform::Filesystem[docker-lv] will propagate my refresh event 2021-04-22T04:59:09.516 Debug: 2021-04-22 04:59:09 +0000 Class[Platform::Lvm::Vg::Cgts_vg]: The container Stage[main] will propagate my refresh event 2021-04-22T04:59:09.520 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:09.536 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/etcd-lv' 2021-04-22T04:59:09.560 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:09.584 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.611 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.632 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): Current: value=2.0, unit=G, kibi=2097152 2021-04-22T04:59:09.638 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): New: value=10.0, unit=G, kibi=10485760 2021-04-22T04:59:09.642 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.653 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 10485760k /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:09.710 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:10.060 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y check /dev/cgts-vg/kubelet-lv' 2021-04-22T04:59:10.166 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/kubelet-lv 10485760k' 2021-04-22T04:59:10.355 Notice: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]/size: size changed '2G' to '10G' 2021-04-22T04:59:10.359 Debug: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]: The container Platform::Filesystem[kubelet-lv] will propagate my refresh event 2021-04-22T04:59:10.364 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.367 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.371 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.376 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.379 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped kubelet-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.382 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv' 2021-04-22T04:59:10.385 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.389 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.394 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs cgts-vg' 2021-04-22T04:59:10.415 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/extension-lv' 2021-04-22T04:59:10.441 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.445 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.457 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped docker-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.466 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv' 2021-04-22T04:59:10.470 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/blkid /dev/cgts-vg/docker-lv' 2021-04-22T04:59:10.476 Debug: 2021-04-22 04:59:10 +0000 Executing: 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1' 2021-04-22T04:59:10.485 Error: 2021-04-22 04:59:10 +0000 Execution of 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1' returned 1: mkfs.xfs: /dev/cgts-vg/docker-lv contains a mounted filesystem 2021-04-22T04:59:10.489 Usage: mkfs.xfs 2021-04-22T04:59:10.492 /* blocksize */ [-b log=n|size=num] Firstly, the resize should not be triggered here, in an older load (2021-04-15 )the docker-lv was 30G before unlock. For some reason docker-lv fs type is drbd and I think it's wrong, it should be xfs: controller-0:/var/log# blkid /dev/cgts-vg/docker-lv /dev/cgts-vg/docker-lv: UUID="c5c72dc8a5af335b" TYPE="drbd" Possible similar bug: https://bugs.launchpad.net/starlingx/+bug/1883825 Test Activity ------------- Developer Testing
2021-04-30 18:27:32 OpenStack Infra starlingx: status In Progress Fix Released
2021-04-30 20:07:42 Ghada Khalil starlingx: importance Undecided High
2021-04-30 20:08:03 Ghada Khalil tags stx.update stx.5.0 stx.cherrypickneeded stx.update
2021-05-01 21:16:27 Ghada Khalil tags stx.5.0 stx.cherrypickneeded stx.update in-r-stx50 stx.5.0 stx.update
2021-06-03 12:39:25 OpenStack Infra tags in-r-stx50 stx.5.0 stx.update in-f-centos8 in-r-stx50 stx.5.0 stx.update
2021-06-07 17:18:08 OpenStack Infra bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=1793527
2021-06-07 17:18:08 OpenStack Infra bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=1819868
2021-06-07 17:18:08 OpenStack Infra bug watch added https://github.com/golang/go/issues/40213
2021-06-07 17:18:08 OpenStack Infra cve linked 2018-15473
2021-06-07 17:18:08 OpenStack Infra cve linked 2019-18634
2021-06-07 17:18:08 OpenStack Infra cve linked 2019-6470
2021-06-07 17:18:08 OpenStack Infra cve linked 2020-13817
2021-06-07 17:18:08 OpenStack Infra cve linked 2020-15705
2021-06-07 17:18:08 OpenStack Infra cve linked 2020-15707
2021-06-07 17:18:08 OpenStack Infra cve linked 2021-3156