All-in-one: host fails after reboot
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Joseph Richard |
Bug Description
Brief Description
-----------------
After rebooting host, it failed to recover with a number of file system failures in console log.
Severity
--------
Major
Steps to Reproduce
------------------
- On simplex or duplex systems.
- sudo reboot -f (or system host-lock/unlock) on a host
- Host fails after reboot
Expected Behavior
------------------
Host reboots and recovers successfully
Actual Behavior
----------------
Host fails to reboot
Reproducibility
---------------
Fairly reproducible. Seen on 2 out of 4 systems.
System Configuration
-------
One node system and Two node systems
Branch/Pull Time/Commit
-------
master as of 2018-08-27_20-18-00
Timestamp/Logs
--------------
[2018-08-27 07:22:04,833] 264 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'
Changed in starlingx: | |
assignee: | nobody → Joseph Richard (josephrichard) |
summary: |
- All-in-one Simplex/Duplex: host fails after reboot + All-in-one: host fails after reboot |
tags: | added: stx.2018.10 stx.config |
Changed in starlingx: | |
status: | New → Triaged |
tags: |
added: stx.1.0 removed: stx.2018.10 |
Adding investigation details by Bob Church:
The global_filter setting on controller-1 is incorrectly set to /dev/disk/ by-path/ pci-0000: 00:17.0- ata-1.0. It needs to include /dev/disk/ by-path/ pci-0000: 00:17.0- ata-3.0 as /dev/sdc is the root disk for this UEFI install and contains all physical volumes for the cgts-volume group. Without it present in the global_filter, the disk is hidden from LVM and will produce the above emergency condition.
controller-1:~# grep global_filter /etc/lvm/lvm.conf global_ filter. disk/by- path/pci- 0000:00: 17.0-ata- 1.0|", "a|/dev/drbd4|", "r|.*|" ] global_ filter.
# Configuration option devices/
# Use global_filter to hide devices from these LVM system components.
# global_filter are not opened by LVM.
global_filter = [ "a|/dev/
# devices/
controller-1:~# ls -Rl /dev/disk/by-path/ 00:17.0- ata-1.0 -> ../../sda 00:17.0- ata-2.0 -> ../../sdb 00:17.0- ata-2.0- part1 -> ../../sdb1 00:17.0- ata-3.0 -> ../../sdc 00:17.0- ata-3.0- part1 -> ../../sdc1 00:17.0- ata-3.0- part2 -> ../../sdc2 00:17.0- ata-3.0- part3 -> ../../sdc3 00:17.0- ata-3.0- part4 -> ../../sdc4 00:17.0- ata-3.0- part5 -> ../../sdc5 00:17.0- ata-3.0- part6 -> ../../sdc6
/dev/disk/by-path/:
total 0
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 9 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
lrwxrwxrwx 1 root root 10 Aug 27 18:33 pci-0000:
After reinstalling the node, setting the root passwd, I was able to drop into the emergency shell and debug. After manually setting the global_filter prior to node unlock, the node successfully booted, applied manifests, and became available.
Lab setup files look ok here: http:// stash.wrs. com/projects/ CGCS/repos/ titanium- tools/browse/ lab/yow/ cgcs-wolfpass- 01_02?at= refs%2Fheads% 2FTC_DEV_ 0003
Looking at the generated hiera data in the node I see the following:
[wrsroot@ controller- 0 hieradata( keystone_ admin)] $ sudo grep -A4 lvm_global_filter 192.168. 204.[34] .yaml 204.3.yaml: openstack: :nova:: storage: :lvm_global_ filter: !!python/unicode '[ "a|/dev/ disk/by- path/pci- 0000:00: 17.0-ata- 1.0|", 204.3.yaml- openstack: :nova:: storage: :lvm_update_ filter: !!python/unicode '[ "a|/dev/ disk/by- path/pci- 0000:00: 17.0-ata- 1.0|", 204.3.yaml- openstack: :nova:: storage: :removing_ pvs: [] 204.4.yaml: openstack: :nova:: storage: :lvm_global_ filter: !!python/unicode '[ "a|/dev/ disk/by- path/pci- 0000:00: 17.0-ata- 1.0|", 204.4.yaml- openstack: :nova:: storage: :lvm_update_ filter: !!python/unicode '[ "a|/dev/ disk/by- path/pci- 0000:00: 17.0-ata- 1.0|", 204.4.yaml- openstack: :nova:: storage: :removing_ pvs: []
192.168.
192.168.204.3.yaml- "a|/dev/drbd4|", "r|.*|" ]'
192.168.
192.168.204.3.yaml- "a|/dev/drbd4|", "r|.*|" ]'
192.168.
--
192.168.
192.168.204.4.yaml- "a|/dev/drbd4|", "r|.*|" ]'
192.168.
192.168.204.4.yaml- "a|/dev/drbd4|", "r|.*|" ]'
192.168.
The filter setting look to be incorrect as it ...