Bug #1789420 “All-in-one: host fails after reboot” : Bugs : StarlingX

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2018-08-28:

#1

Download full text (12.2 KiB)

Adding investigation details by Bob Church:

The global_filter setting on controller-1 is incorrectly set to /dev/disk/by-path/pci-0000:00:17.0-ata-1.0. It needs to include /dev/disk/by-path/pci-0000:00:17.0-ata-3.0 as /dev/sdc is the root disk for this UEFI install and contains all physical volumes for the cgts-volume group. Without it present in the global_filter, the disk is hidden from LVM and will produce the above emergency condition.

controller-1:~# grep global_filter /etc/lvm/lvm.conf
        # Configuration option devices/global_filter.
        # Use global_filter to hide devices from these LVM system components.
        # global_filter are not opened by LVM.
    global_filter = [ "a|/dev/disk/by-path/pci-0000:00:17.0-ata-1.0|", "a|/dev/drbd4|", "r|.*|" ]
        # devices/global_filter.

controller-1:~# ls -Rl /dev/disk/by-path/
/dev/disk/by-path/:
total 0
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-1.0 -> ../../sda
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-2.0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-2.0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-3.0 -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part4 -> ../../sdc4
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part5 -> ../../sdc5
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part6 -> ../../sdc6

After reinstalling the node, setting the root passwd, I was able to drop into the emergency shell and debug. After manually setting the global_filter prior to node unlock, the node successfully booted, applied manifests, and became available.

Lab setup files look ok here: http://stash.wrs.com/projects/CGCS/repos/titanium-tools/browse/lab/yow/cgcs-wolfpass-01_02?at=refs%2Fheads%2FTC_DEV_0003

Looking at the generated hiera data in the node I see the following:

The filter setting look to be incorrect as it ...

Adding investigation details by Bob Church:

The global_filter setting on controller-1 is incorrectly set to /dev/disk/by-path/pci-0000:00:17.0-ata-1.0. It needs to include /dev/disk/by-path/pci-0000:00:17.0-ata-3.0 as /dev/sdc is the root disk for this UEFI install and contains all physical volumes for the cgts-volume group. Without it present in the global_filter, the disk is hidden from LVM and will produce the above emergency condition.

controller-1:~# grep global_filter /etc/lvm/lvm.conf 
        # Configuration option devices/global_filter. 
        # Use global_filter to hide devices from these LVM system components. 
        # global_filter are not opened by LVM. 
    global_filter = [ "a|/dev/disk/by-path/pci-0000:00:17.0-ata-1.0|", "a|/dev/drbd4|", "r|.*|" ] 
        # devices/global_filter.

controller-1:~# ls -Rl /dev/disk/by-path/ 
/dev/disk/by-path/: 
total 0 
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-1.0 -> ../../sda 
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-2.0 -> ../../sdb 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-2.0-part1 -> ../../sdb1 
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:00:17.0-ata-3.0 -> ../../sdc 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part1 -> ../../sdc1 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part2 -> ../../sdc2 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part3 -> ../../sdc3 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part4 -> ../../sdc4 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part5 -> ../../sdc5 
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:00:17.0-ata-3.0-part6 -> ../../sdc6

After reinstalling the node, setting the root passwd, I was able to drop into the emergency shell and debug. After manually setting the global_filter prior to node unlock, the node successfully booted, applied manifests, and became available.

Lab setup files look ok here: http://stash.wrs.com/projects/CGCS/repos/titanium-tools/browse/lab/yow/cgcs-wolfpass-01_02?at=refs%2Fheads%2FTC_DEV_0003

Looking at the generated hiera data in the node I see the following:

The filter setting look to be incorrect as it should contain /dev/disk/by-path/pci-0000:00:17.0-ata-3.0 (/dev/sdc). /dev/disk/by-path/pci-0000:00:17.0-ata-1.0 corresponds to the disk being specified for nova-local in lab_setup settings: 
CONTROLLER0_LOCAL_STORAGE="local_image|/dev/disk/by-path/pci-0000:00:17.0-ata-1.0|fixed|50" 
CONTROLLER1_LOCAL_STORAGE="local_image|/dev/disk/by-path/pci-0000:00:17.0-ata-1.0|fixed|50"

The problem stems from this commit: Dynamically load existing puppet plugins:  https://review.openstack.org/#/c/594507/

Specifically, this sets an order in which the plugins run. We have a dependency between the storage plugin and the nova plugin as they both will touch the LVM global_filter setting. The storage plugin needs to run before the nova plugin. With the following change the node consistently unlocks and no longer drops into emergency mode on reboot.

NOTE: To test this, I have manually made this change on controller-0 (yow-cgcs-wolfpass-1-2): /usr/lib64/python2.7/site-packages/sysinv-1.0-py2.7.egg-info/entry_points.txt.

1 file changed, 12 insertions(+), 12 deletions(-)
sysinv/sysinv/sysinv/setup.cfg | 24 ++++++++++++------------

modified   sysinv/sysinv/sysinv/setup.cfg
@@ -50,18 +50,18 @@ systemconfig.puppet_plugins =
     010_nfv = sysinv.puppet.nfv:NfvPuppet
     011_ceph = sysinv.puppet.ceph:CephPuppet
     012_device = sysinv.puppet.device:DevicePuppet
-    013_nova = sysinv.puppet.nova:NovaPuppet
-    014_neutron = sysinv.puppet.neutron:NeutronPuppet
-    015_horizon = sysinv.puppet.horizon:HorizonPuppet
-    016_glance = sysinv.puppet.glance:GlancePuppet
-    017_gnocchi = sysinv.puppet.gnocchi:GnocchiPuppet
-    018_cinder = sysinv.puppet.cinder:CinderPuppet
-    019_ceilometer = sysinv.puppet.ceilometer:CeilometerPuppet
-    020_aodh = sysinv.puppet.aodh:AodhPuppet
-    021_heat = sysinv.puppet.heat:HeatPuppet
-    022_magnum = sysinv.puppet.magnum:MagnumPuppet
-    023_murano = sysinv.puppet.murano:MuranoPuppet
-    024_storage = sysinv.puppet.storage:StoragePuppet
+    013_storage = sysinv.puppet.storage:StoragePuppet
+    014_nova = sysinv.puppet.nova:NovaPuppet
+    015_neutron = sysinv.puppet.neutron:NeutronPuppet
+    016_horizon = sysinv.puppet.horizon:HorizonPuppet
+    017_glance = sysinv.puppet.glance:GlancePuppet
+    018_gnocchi = sysinv.puppet.gnocchi:GnocchiPuppet
+    019_cinder = sysinv.puppet.cinder:CinderPuppet
+    020_ceilometer = sysinv.puppet.ceilometer:CeilometerPuppet
+    021_aodh = sysinv.puppet.aodh:AodhPuppet
+    022_heat = sysinv.puppet.heat:HeatPuppet
+    023_magnum = sysinv.puppet.magnum:MagnumPuppet
+    024_murano = sysinv.puppet.murano:MuranoPuppet
     025_ironic = sysinv.puppet.ironic:IronicPuppet
     026_panko = sysinv.puppet.panko:PankoPuppet
     027_dcmanager = sysinv.puppet.dcmanager:DCManagerPuppet

Prior to this change for stevedore plugin support. The following shows the plugin order for the unlock based on personality type:

-    def update_controller_config(self, host, config_uuid=None):
-        """Update the configuration for a specific controller host"""
-        try:
-            # NOTE: order is important due to cached context data
-            config = {}
-            config.update(self.platform.get_host_config(host, config_uuid))
-            config.update(self.interface.get_host_config(host))
-            config.update(self.ovs.get_host_config(host))
-            config.update(self.networking.get_host_config(host))
-            config.update(self.storage.get_host_config(host))
-            config.update(self.ldap.get_host_config(host))
-            config.update(self.nfv.get_host_config(host))
-            config.update(self.ceph.get_host_config(host))
-            config.update(self.cinder.get_host_config(host))
-            config.update(self.device.get_host_config(host))
-            config.update(self.nova.get_host_config(host))
-            config.update(self.neutron.get_host_config(host))
-            config.update(self.smapi.get_host_config(host))
-            config.update(self.fm.get_host_config(host))
-            # service_parameter must be last to permit overrides
-            config.update(self.service_parameter.get_host_config(host))
-
-            return config
-        except Exception:
-            LOG.exception("failed to create host config: %s" % host.uuid)
-            raise
-
-    def update_compute_config(self, host, config_uuid=None):
-        """Update the configuration for a specific compute host"""
-        try:
-            # NOTE: order is important due to cached context data
-            config = {}
-            config.update(self.platform.get_host_config(host, config_uuid))
-            config.update(self.interface.get_host_config(host))
-            config.update(self.ovs.get_host_config(host))
-            config.update(self.networking.get_host_config(host))
-            config.update(self.storage.get_host_config(host))
-            config.update(self.ceph.get_host_config(host))
-            config.update(self.device.get_host_config(host))
-            config.update(self.nova.get_host_config(host))
-            config.update(self.neutron.get_host_config(host))
-            config.update(self.kubernetes.get_host_config(host))
-            config.update(self.ldap.get_host_config(host))
-            # service_parameter must be last to permit overrides
-            config.update(self.service_parameter.get_host_config(host))
-
-            return config
-        except Exception:
-            LOG.exception("failed to create host config: %s" % host.uuid)
-            raise
-
-    def update_storage_config(self, host, config_uuid=None):
-        """Update the configuration for a specific storage host"""
-        try:
-            # NOTE: order is important due to cached context data
-            config = {}
-            config.update(self.platform.get_host_config(host, config_uuid))
-            config.update(self.interface.get_host_config(host))
-            config.update(self.networking.get_host_config(host))
-            config.update(self.storage.get_host_config(host))
-            config.update(self.ceph.get_host_config(host))
-            config.update(self.ldap.get_host_config(host))
-            # service_parameter must be last to permit overrides
-            config.update(self.service_parameter.get_host_config(host))
-
-            return config
-        except Exception:
-            LOG.exception("failed to create host config: %s" % host.uuid)
-            raise
-

I believe that this will only be a problem on installs where nova-local is on a separate disk than the cgts-vg (i.e. a 3 disk install). Unless this config was specifically tested, I can see why this may not have been caught and why it’s not seen everywhere.

With the above said, MTC is still rebooting controller-1. I have locked controller-1 on yow-cgcs-wolfpass-1-2. I have the lab reserved with a reservation note so it won’t be reinstalled. From the looks of it, the reboot is occurring because the compute function is never getting enabled. I see the following recurring in sysinv.log:

2018-08-28 04:30:47.172 251060 CRITICAL neutron [req-ca2cb4ca-8014-4362-a74d-7113a072ecc3 - - - - -] Unhandled error: AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details se.

Because of this error in sysinv, the state of controller-1 storage is not being reported to controller-0. All configured storage for controller-1 in the inventory is still in transitional adding states, but are present on controller-1 after all the manifests are run:

controller-1:/var/log# vgs 
  VG         #PV #LV #SN Attr   VSize    VFree  
  cgts-vg      2  10   0 wz--n- <285.97g <12.16g
  nova-local   1   1   0 wz--n- <447.13g      0 
controller-1:/var/log# pvs
  PV         VG         Fmt  Attr PSize    PFree 
  /dev/sda   nova-local lvm2 a--  <447.13g     0 
  /dev/sdc4  cgts-vg    lvm2 a--   147.00g <7.19g
  /dev/sdc5  cgts-vg    lvm2 a--  <138.97g <4.97g
controller-1:/var/log# lvs
  LV                 VG         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  backup-lv          cgts-vg    -wi-ao----   96.00g                                                    
  cgcs-lv            cgts-vg    -wi-ao----   38.00g                                                    
  extension-lv       cgts-vg    -wi-ao----    1.00g                                                    
  gnocchi-lv         cgts-vg    -wi-ao----    5.00g                                                    
  img-conversions-lv cgts-vg    -wi-ao----   38.00g                                                    
  log-lv             cgts-vg    -wi-ao----    7.81g                                                    
  pgsql-lv           cgts-vg    -wi-ao----   76.00g                                                    
  platform-lv        cgts-vg    -wi-ao----    2.00g                                                    
  rabbit-lv          cgts-vg    -wi-ao----    2.00g                                                    
  scratch-lv         cgts-vg    -wi-ao----    8.00g                                                    
  instances_lv       nova-local -wi-ao---- <447.13g

Changed in starlingx:
importance:	Undecided → High

Joseph Richard (josephrichard) on 2018-08-28

Changed in starlingx:
assignee:	nobody → Joseph Richard (josephrichard)

Ghada Khalil (gkhalil) on 2018-08-28

summary:	- All-in-one Simplex/Duplex: host fails after reboot + All-in-one: host fails after reboot
tags:	added: stx.2018.10 stx.config

Ghada Khalil (gkhalil) on 2018-08-28

Changed in starlingx:
status:	New → Triaged

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-08-29: Fix proposed to stx-config (master)

#2

Fix proposed to branch: master
Review: https://review.openstack.org/597698

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2018-08-31:

#3

Manually marking as Fix Released; not sure why it was done automatically by gerrit when the commit was merged.

Changed in starlingx:
status:	In Progress → Fix Released

Ken Young (kenyis) on 2019-04-06

tags:

added: stx.1.0
removed: stx.2018.10

StarlingX

All-in-one: host fails after reboot

Bug Description

Other bug subscribers

Remote bug watches