Controller-0 configuration failed after initial unlock

[Yvonne Ding]

Brief Description
-----------------

Controller-0 configuration is failed after initial unlock. The error of "device minor not allocated" is found for drbd-platform. Depndency
Kmod::Load[vfio-pci] for Exec[vfio-iommu-mode] at /usr/share/puppet/modules/platform/manifests/vswitch.pp:16 can't be found.

Platform::Drbd::Filesystem[drbd-platform]/Drbd::Resource[drbd-platform]/Drbd::Resource::Enable[drbd-platform]/Drbd::Resource::Up[drbd-platform]/Exec[initialize DRBD metadata for drbd-platform]/unless: 2: Failure: (127) Device minor not allocated

Severity
--------
Major

Steps to Reproduce
------------------
1. Install and configure controller-0 in a stx-openstack system
2. Unlock controller-0
3. controller-0 config failed after unlock

TC-name:

Expected Behavior
-----------------
Controller-0 config succeed after unlock

Actual Behavior
----------------
Controller-0 config failed after unlock

Reproducibility
---------------
first attempt

System Configuration
--------------------
AIO-DX

Lab-name:
ip-5-6

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20200710T013452Z"

Timestamp/Logs
--------------
1) controller-0 unlocked:

| 2020-07-13T04:28: | log | 200.022 | controller-0 is now 'offline' | host=controller-0.status= | not-applicable |

| 15.072410 | | | | offline | |

| | | | | | |

| 2020-07-13T04:28: | log | 200.021 | controller-0 manual 'unlock' request | host=controller-0.command= | not-applicable |

| 14.989399 | | | | unlock | |

2) controller-0 was unlocked and enabled, but rebooted shortly after:

| | | | | | |

| 2020-07-13T04:39: | set | 200.011 | controller-0 experienced a configuration failure. | host=controller-0 | critical |

| 39.891498 | | | | | |

| | | | | | |

| 2020-07-13T04:39: | log | 200.022 | controller-0 is now 'disabled' | host=controller-0.state= | not-applicable |

| 39.798847 | | | | disabled | |

| | | | | | |

| 2020-07-13T04:39: | log | 200.022 | controller-0 is now 'enabled' | host=controller-0.state= | not-applicable |

| 30.248938 | | | | enabled | |

| | | | | | |

| 2020-07-13T04:39: | log | 200.022 | controller-0 is now 'online' | host=controller-0.status= | not-applicable |

| 29.755892 | | | | online | |

3) puppet logs
/var/log/puppet/2020-07-13-04-34-16_controller/puppet.log:2020-07-13T04:36:36.547 Debug: 2020-07-13 04:36:36 +0000 /Stage[main]/Platform::Drbd::Pgsql/Platform::Drbd::Filesystem[drbd-pgsql]/Drbd::Resource[drbd-pgsql]/Drbd::Resource::Enable[drbd-pgsql]/Drbd::Resource::Up[drbd-pgsql]/Exec[initialize DRBD metadata for drbd-pgsql]/unless: 0: Failure: (127) Device minor not allocated

/var/log/puppet/2020-07-13-04-34-16_controller/puppet.log:2020-07-13T04:36:44.825 echo "Error removing osd ${osd_id}, exit code: ${RET}"

/var/log/puppet/2020-07-13-04-34-16_controller/puppet.log:2020-07-13T04:36:44.850 echo "Error creating osd ${osd_id}, exit code: ${RET}"

/var/log/puppet/2020-07-13-04-34-16_controller/puppet.log:2020-07-13T04:36:44.927 echo "Error removing osd ${osd_id}, exit code: ${RET}"

/var/log/puppet/2020-07-13-04-34-16_controller/puppet.log:2020-07-13T04:36:44.952 echo "Error creating osd ${osd_id}, exit code: ${RET}"

/var/log/puppet/2020-07-13-04-39-24_worker/puppet.log:2020-07-13T04:39:26.587 Debug: 2020-07-13 04:39:26 +0000 Failed to load library 'cfpropertylist' for feature 'cfpropertylist'

/var/log/puppet/2020-07-13-04-39-24_worker/puppet.log:2020-07-13T04:39:26.732 Debug: 2020-07-13 04:39:26 +0000 Failed to load library 'cfpropertylist' for feature 'cfpropertylist'

/var/log/puppet/2020-07-13-04-39-24_worker/puppet.log:2020-07-13T04:39:36.097 Error: 2020-07-13 04:39:35 +0000 Could not find dependency Kmod::Load[vfio-pci] for Exec[vfio-iommu-mode] at /usr/share/puppet/modules/platform/manifests/vswitch.pp:16

logs of .tar
https://files.starlingx.kube.cengn.ca/launchpad/1887409

Test Activity
-------------
stx-openstack is applied

[Ghada Khalil (gkhalil)]

Jim, can you please see if the disk support has been dropped from the centos 8 kernel for this old h/w

[Jim Somerville (jsomervi)]

It's grey zone. The isci driver is there and we have configured it in, but RH lists it as not supported in 8. If it didn't work, the install would have bombed out early.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/hardware-enablement_considerations-in-adopting-rhel-8

[Ghada Khalil (gkhalil)]

Marking as low priority given that this is old hardware that is not officially supported in CentOS 8.

[Jim Somerville (jsomervi)]

This is an AIO-DX system (as reported) yet only logs for one controller were supplied.