Investigation so far:-

Cinder error:-

2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'



Good:-

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/home/zuul/ansible.log

~~~
2020-04-14 02:06:41,931 p=17519 u=root |  TASK [cinder create LVM volume group] ******************************************
2020-04-14 02:06:41,932 p=17519 u=root |  Tuesday 14 April 2020  02:06:41 +0000 (0:00:00.333)       0:01:17.941 ********* 
2020-04-14 02:06:42,599 p=17519 u=root |  changed: [standalone]
~~~

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/var/log/extra/lvm.txt

~~~
  VG             #PV #LV #SN Attr   VSize   VFree  
  cinder-volumes   1   1   0 wz--n- <10.04g 484.00m
  PV         VG             Fmt  Attr PSize   PFree  
  /dev/loop2 cinder-volumes lvm2 a--  <10.04g 484.00m
  LV                  VG             Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  cinder-volumes-pool cinder-volumes twi-aotz-- <9.54g             0.00   10.58                           

~~~

https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/718545/19/check/tripleo-ci-centos-8-standalone/ab81b23/logs/undercloud/var/log/journal.txt
~~~
Apr 14 02:06:42 standalone.localdomain ansible-command[20051]: Invoked with creates=/dev/loop2 executable=/bin/bash _raw_params=if ! losetup /dev/loop2; then
                                                                 losetup /dev/loop2 /var/lib/cinder/cinder-volumes
                                                               fi
                                                               if ! pvdisplay | grep cinder-volumes; then
                                                                 pvcreate /dev/loop2
                                                               fi
                                                               if ! vgdisplay | grep cinder-volumes; then
                                                                 vgcreate cinder-volumes /dev/loop2
                                                               fi
                                                                _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None removes=None stdin=None
Apr 14 02:06:42 standalone.localdomain kernel: loop: module loaded
Apr 14 02:06:42 standalone.localdomain systemd[1]: Created slice system-lvm2\x2dpvscan.slice.
Apr 14 02:06:42 standalone.localdomain systemd[1]: Starting LVM event activation on device 7:2...
Apr 14 02:06:42 standalone.localdomain lvm[20065]:   pvscan[20065] PV /dev/loop2 not used.
~~~



Bad:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/home/zuul/ansible.log

~~~
2020-04-14 12:43:13,479 p=28349 u=root |  TASK [cinder create LVM volume group] ******************************************
2020-04-14 12:43:13,479 p=28349 u=root |  Tuesday 14 April 2020  12:43:13 +0000 (0:00:00.436)       0:01:34.648 ********* 
2020-04-14 12:43:13,739 p=28349 u=root |  ok: [standalone]   ------> ok instead of changed(comparing good run)
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/extra/lvm.txt

blank


https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/journal.txt


~~~
Apr 14 12:43:13 standalone.localdomain ansible-command[30990]: Invoked with creates=/dev/loop2 executable=/bin/bash _raw_params=if ! losetup /dev/loop2; then
                                                                 losetup /dev/loop2 /var/lib/cinder/cinder-volumes
                                                               fi
                                                               if ! pvdisplay | grep cinder-volumes; then
                                                                 pvcreate /dev/loop2
                                                               fi
                                                               if ! vgdisplay | grep cinder-volumes; then
                                                                 vgcreate cinder-volumes /dev/loop2
                                                               fi
                                                                _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None 
~~~




These logs are missing in bad run:-
~~~
Apr 14 02:06:42 standalone.localdomain kernel: loop: module loaded
Apr 14 02:06:42 standalone.localdomain systemd[1]: Created slice system-lvm2\x2dpvscan.slice.
Apr 14 02:06:42 standalone.localdomain systemd[1]: Starting LVM event activation on device 7:2...
Apr 14 02:06:42 standalone.localdomain lvm[20065]:   pvscan[20065] PV /dev/loop2 not used.
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ea/715006/8/gate/tripleo-ci-centos-8-standalone/3ea0819/logs/undercloud/var/log/extra/lsmod.txt --

loop module didn't load confirmed from lsmod output.. good run have it


It seems like https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/cinder/cinder-common-container-puppet.yaml#L122 was not completed and somehow that file exists, so task didn't got triggered