Comment 1 for bug 1872881

Investigation so far:-

Cinder error:-

2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'

Good:-

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/home/zuul/ansible.log

~~~
2020-04-14 02:06:41,931 p=17519 u=root | TASK [cinder create LVM volume group] ******************************************
2020-04-14 02:06:41,932 p=17519 u=root | Tuesday 14 April 2020 02:06:41 +0000 (0:00:00.333) 0:01:17.941 *********
2020-04-14 02:06:42,599 p=17519 u=root | changed: [standalone]
~~~

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/var/log/extra/lvm.txt

~~~
  VG #PV #LV #SN Attr VSize VFree
  cinder-volumes 1 1 0 wz--n- <10.04g 484.00m
  PV VG Fmt Attr PSize PFree
  /dev/loop2 cinder-volumes lvm2 a-- <10.04g 484.00m
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-aotz-- <9.54g 0.00 10.58

~~~

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/var/log/journal.txt
~~~
Apr 14 02:06:42 standalone.localdomain ansible-command[20051]: Invoked with creates=/dev/loop2 executable=/bin/bash _raw_params=if ! losetup /dev/loop2; then
                                                                 losetup /dev/loop2 /var/lib/cinder/cinder-volumes
                                                               fi
                                                               if ! pvdisplay | grep cinder-volumes; then
                                                                 pvcreate /dev/loop2
                                                               fi
                                                               if ! vgdisplay | grep cinder-volumes; then
                                                                 vgcreate cinder-volumes /dev/loop2
                                                               fi
                                                                _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None removes=None stdin=None
Apr 14 02:06:42 standalone.localdomain kernel: loop: module loaded
Apr 14 02:06:42 standalone.localdomain systemd[1]: Created slice system-lvm2\x2dpvscan.slice.
Apr 14 02:06:42 standalone.localdomain systemd[1]: Starting LVM event activation on device 7:2...
Apr 14 02:06:42 standalone.localdomain lvm[20065]: pvscan[20065] PV /dev/loop2 not used.
~~~

Bad:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/home/zuul/ansible.log

~~~
2020-04-14 12:43:13,479 p=28349 u=root | TASK [cinder create LVM volume group] ******************************************
2020-04-14 12:43:13,479 p=28349 u=root | Tuesday 14 April 2020 12:43:13 +0000 (0:00:00.436) 0:01:34.648 *********
2020-04-14 12:43:13,739 p=28349 u=root | ok: [standalone] ------> ok instead of changed(comparing good run)
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/extra/lvm.txt

blank

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/journal.txt

~~~
Apr 14 12:43:13 standalone.localdomain ansible-command[30990]: Invoked with creates=/dev/loop2 executable=/bin/bash _raw_params=if ! losetup /dev/loop2; then
                                                                 losetup /dev/loop2 /var/lib/cinder/cinder-volumes
                                                               fi
                                                               if ! pvdisplay | grep cinder-volumes; then
                                                                 pvcreate /dev/loop2
                                                               fi
                                                               if ! vgdisplay | grep cinder-volumes; then
                                                                 vgcreate cinder-volumes /dev/loop2
                                                               fi
                                                                _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None
~~~

These logs are missing in bad run:-
~~~
Apr 14 02:06:42 standalone.localdomain kernel: loop: module loaded
Apr 14 02:06:42 standalone.localdomain systemd[1]: Created slice system-lvm2\x2dpvscan.slice.
Apr 14 02:06:42 standalone.localdomain systemd[1]: Starting LVM event activation on device 7:2...
Apr 14 02:06:42 standalone.localdomain lvm[20065]: pvscan[20065] PV /dev/loop2 not used.
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/extra/lsmod.txt --

loop module didn't load confirmed from lsmod output.. good run have it

It seems like https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cinder/cinder-common-container-puppet.yaml#L122 was not completed and somehow that file exists, so task didn't got triggered