Fuel for OpenStack

Booting bootstrap without any Disc Fuel 5.0.1

Bug #1340207 reported by Tony Tarasov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Vladimir Kozhukalov
Fuel for OpenStack 5.1
5.0.x
Invalid
High
Vladimir Kozhukalov
Fuel for OpenStack 5.0.2

Bug Description

We using Fuel 5.0.1 with zabbix implementation

We have the following Hardware:
2 X 146 SAS on raid 1

When we trying to boot some node in bootstrap mode, it coming without any physical disc

See the following for help:

[root@bootstrap ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom

[root@bootstrap ~]# for dev in $(ls /sys/block); do udevadm info --query=property --export --name=${dev}; done
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop0'
MAJOR='7'
MINOR='0'
DEVNAME='/dev/loop0'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:0'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop1'
MAJOR='7'
MINOR='1'
DEVNAME='/dev/loop1'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:1'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop2'
MAJOR='7'
MINOR='2'
DEVNAME='/dev/loop2'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:2'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop3'
MAJOR='7'
MINOR='3'
DEVNAME='/dev/loop3'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:3'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop4'
MAJOR='7'
MINOR='4'
DEVNAME='/dev/loop4'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:4'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop5'
MAJOR='7'
MINOR='5'
DEVNAME='/dev/loop5'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:5'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop6'
MAJOR='7'
MINOR='6'
DEVNAME='/dev/loop6'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:6'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/loop7'
MAJOR='7'
MINOR='7'
DEVNAME='/dev/loop7'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/7:7'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram0'
MAJOR='1'
MINOR='0'
DEVNAME='/dev/ram0'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:0'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram1'
MAJOR='1'
MINOR='1'
DEVNAME='/dev/ram1'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:1'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram10'
MAJOR='1'
MINOR='10'
DEVNAME='/dev/ram10'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:10'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram11'
MAJOR='1'
MINOR='11'
DEVNAME='/dev/ram11'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:11'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram12'
MAJOR='1'
MINOR='12'
DEVNAME='/dev/ram12'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:12'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram13'
MAJOR='1'
MINOR='13'
DEVNAME='/dev/ram13'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:13'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram14'
MAJOR='1'
MINOR='14'
DEVNAME='/dev/ram14'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:14'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram15'
MAJOR='1'
MINOR='15'
DEVNAME='/dev/ram15'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:15'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram2'
MAJOR='1'
MINOR='2'
DEVNAME='/dev/ram2'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:2'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram3'
MAJOR='1'
MINOR='3'
DEVNAME='/dev/ram3'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:3'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram4'
MAJOR='1'
MINOR='4'
DEVNAME='/dev/ram4'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:4'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram5'
MAJOR='1'
MINOR='5'
DEVNAME='/dev/ram5'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:5'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram6'
MAJOR='1'
MINOR='6'
DEVNAME='/dev/ram6'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:6'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram7'
MAJOR='1'
MINOR='7'
DEVNAME='/dev/ram7'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:7'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram8'
MAJOR='1'
MINOR='8'
DEVNAME='/dev/ram8'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:8'
UDEV_LOG='3'
DEVPATH='/devices/virtual/block/ram9'
MAJOR='1'
MINOR='9'
DEVNAME='/dev/ram9'
DEVTYPE='disk'
SUBSYSTEM='block'
DEVLINKS='/dev/block/1:9'
UDEV_LOG='3'
DEVPATH='/devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.0/host2/target2:0:0/2:0:0:0/block/sda'
MAJOR='8'
MINOR='0'
DEVNAME='/dev/sda'
DEVTYPE='disk'
SUBSYSTEM='block'
ID_VENDOR='iDRAC'
ID_VENDOR_ENC='iDRAC\x20\x20\x20'
ID_VENDOR_ID='046b'
ID_MODEL='LCDRIVE'
ID_MODEL_ENC='LCDRIVE\x20\x20\x20\x20\x20\x20\x20\x20\x20'
ID_MODEL_ID='ff90'
ID_REVISION='1.00'
ID_SERIAL='iDRAC_LCDRIVE_serial-0:0'
ID_SERIAL_SHORT='serial'
ID_TYPE='disk'
ID_INSTANCE='0:0'
ID_BUS='usb'
ID_USB_INTERFACES=':080650:'
ID_USB_INTERFACE_NUM='00'
ID_USB_DRIVER='usb-storage'
ID_PATH='pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0'
DEVLINKS='/dev/block/8:0 /dev/disk/by-id/usb-iDRAC_LCDRIVE_serial-0:0 /dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0'
UDEV_LOG='3'
DEVPATH='/devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.2/host4/target4:0:0/4:0:0:0/block/sdb'
MAJOR='8'
MINOR='16'
DEVNAME='/dev/sdb'
DEVTYPE='disk'
SUBSYSTEM='block'
ID_VENDOR='iDRAC'
ID_VENDOR_ENC='iDRAC\x20\x20\x20'
ID_VENDOR_ID='046b'
ID_MODEL='Virtual_Floppy'
ID_MODEL_ENC='Virtual\x20Floppy\x20\x20'
ID_MODEL_ID='ff90'
ID_REVISION='1.00'
ID_SERIAL='iDRAC_Virtual_Floppy_serial-0:0'
ID_SERIAL_SHORT='serial'
ID_TYPE='disk'
ID_INSTANCE='0:0'
ID_BUS='usb'
ID_USB_INTERFACES=':080650:'
ID_USB_INTERFACE_NUM='02'
ID_USB_DRIVER='usb-storage'
ID_PATH='pci-0000:00:1d.7-usb-0:3:1.2-scsi-0:0:0:0'
DEVLINKS='/dev/block/8:16 /dev/disk/by-id/usb-iDRAC_Virtual_Floppy_serial-0:0 /dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.2-scsi-0:0:0:0'
UDEV_LOG='3'
DEVPATH='/devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.1/host3/target3:0:0/3:0:0:0/block/sr0'
MAJOR='11'
MINOR='0'
DEVNAME='/dev/sr0'
DEVTYPE='disk'
SUBSYSTEM='block'
ID_CDROM='1'
ID_VENDOR='iDRAC'
ID_VENDOR_ENC='iDRAC\x20\x20\x20'
ID_VENDOR_ID='046b'
ID_MODEL='Virtual_CDROM'
ID_MODEL_ENC='Virtual\x20CDROM\x20\x20\x20'
ID_MODEL_ID='ff90'
ID_REVISION='1.00'
ID_SERIAL='iDRAC_Virtual_CDROM_serial-0:0'
ID_SERIAL_SHORT='serial'
ID_TYPE='cd'
ID_INSTANCE='0:0'
ID_BUS='usb'
ID_USB_INTERFACES=':080650:'
ID_USB_INTERFACE_NUM='01'
ID_USB_DRIVER='usb-storage'
ID_PATH='pci-0000:00:1d.7-usb-0:3:1.1-scsi-0:0:0:0'
ACL_MANAGE='1'
GENERATED='1'
DEVLINKS='/dev/block/11:0 /dev/scd0 /dev/disk/by-id/usb-iDRAC_Virtual_CDROM_serial-0:0 /dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.1-scsi-0:0:0:0 /dev/cdrom'

I attached the picture

Revision history for this message
Tony Tarasov (atarasov) wrote :
Revision history for this message
Tony Tarasov (atarasov) wrote :

Some information about Hardware: Dell M610 Blade

Nastya Urlapova (aurlapova)
Changed in fuel:
importance: Undecided → Medium
assignee: nobody → Fuel Python Team (fuel-python)
milestone: none → 5.0.1
Dmitry Borodaenko (angdraug)
tags: added: customer-found
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Miroslav, Vladimir Kozhukalov, can you please provide your thoughts on why it doesn't see disks? Agent filters them out?

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Miroslav Anashkin (manashkin)
milestone: 5.0.1 → 5.1
Revision history for this message
Tony Tarasov (atarasov) wrote :

There is the accurate model of RAID controller form our customer:

Dell M610 blade used SAS6IR controller. Below is the link for downloading the selected based on Redhat linux 6.

http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=G5TG7&fileId=3126014167&osCode=RH60&productCode=poweredge-m610&languageCode=EN&categoryId=SF

Entire driver kit for Dell M610
http://www.dell.com/support/home/us/en/04/Drivers/DriversDetails?driverId=38NVT&fileId=3325063955&osCode=RH60&productCode=poweredge-m610&languageCode=EN&categoryId=DD

Hope this helps you , do let me know for any assistance on this.

Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

As far as I can see from udevadm output system successfully detects hard drive sda
UDEV_LOG='3'
DEVPATH='/devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.0/host2/target2:0:0/2:0:0:0/block/sda'
MAJOR='8'
MINOR='0'
DEVNAME='/dev/sda'
DEVTYPE='disk'
SUBSYSTEM='block'
ID_VENDOR='iDRAC'
ID_VENDOR_ENC='iDRAC\x20\x20\x20'
ID_VENDOR_ID='046b'
ID_MODEL='LCDRIVE'
ID_MODEL_ENC='LCDRIVE\x20\x20\x20\x20\x20\x20\x20\x20\x20'
ID_MODEL_ID='ff90'
ID_REVISION='1.00'
ID_SERIAL='iDRAC_LCDRIVE_serial-0:0'
ID_SERIAL_SHORT='serial'
ID_TYPE='disk'
ID_INSTANCE='0:0'
ID_BUS='usb'
ID_USB_INTERFACES=':080650:'
ID_USB_INTERFACE_NUM='00'
ID_USB_DRIVER='usb-storage'
ID_PATH='pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0'
DEVLINKS='/dev/block/8:0 /dev/disk/by-id/usb-iDRAC_LCDRIVE_serial-0:0 /dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0'

Im dubious about the driver incompatibility. Maybe system exposes /dev/sda as removable drive? If so, the problem could be solved by adding this hard drive model in the list of devices which are exposed as removable (https://github.com/stackforge/fuel-web/blob/master/bin/agent#L45). Please, provide logs from the discovery agent so as to make it possible to figure out exactly whether the problem is just a removable bit or it is about driver incompatibility.

Revision history for this message
Tony Tarasov (atarasov) wrote :
Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

As I can see in logs disk sda has removable bit

2014-07-10T10:12:44.994631+00:00 debug: 10:16:08.146646 #7465] DEBUG -- : Getting udev properties for device: sda
2014-07-10T10:12:44.994780+00:00 debug: 10:16:08.149025 #7465] DEBUG -- : Device sda udev properties: {"ID_USB_INTERFACE_NUM"=>"00",
 "MAJOR"=>"8", "UDEV_LOG"=>"3", "ID_USB_DRIVER"=>"usb-storage", "ID_USB_INTERFACES"=>":080650:", "ID_VENDOR_ID"=>"046b", "ID_VENDOR"
=>"iDRAC", "SUBSYSTEM"=>"block", "DEVTYPE"=>"disk", "DEVPATH"=>"/devices/pci0000:00/0000:00:1d.7/usb2/2-3/2-3:1.0/host2/target2:0:0/
2:0:0:0/block/sda", "ID_TYPE"=>"disk", "ID_REVISION"=>"1.00", "ID_VENDOR_ENC"=>"iDRAC\\x20\\x20\\x20", "DEVLINKS"=>"/dev/block/8:0 /
dev/disk/by-id/usb-iDRAC_LCDRIVE_serial-0:0 /dev/disk/by-path/pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0", "ID_BUS"=>"usb", "ID_INSTA
NCE"=>"0:0", "ID_MODEL"=>"LCDRIVE", "ID_PATH"=>"pci-0000:00:1d.7-usb-0:3:1.0-scsi-0:0:0:0", "ID_MODEL_ID"=>"ff90", "MINOR"=>"0", "ID
_SERIAL_SHORT"=>"serial", "ID_SERIAL"=>"iDRAC_LCDRIVE_serial-0:0", "ID_MODEL_ENC"=>"LCDRIVE\\x20\\x20\\x20\\x20\\x20\\x20\\x20\\x20\
\x20", "DEVNAME"=>"/dev/sda"}
2014-07-10T10:12:44.994879+00:00 debug: 10:16:08.149066 #7465] DEBUG -- : Trying to find out if device sda is removable or not
2014-07-10T10:12:44.994987+00:00 debug: 10:16:08.149138 #7465] DEBUG -- : Device sda removable parameter: "1"

By default we filter out removable devices in discovery agent https://github.com/stackforge/fuel-web/blob/master/bin/agent#L262 and as I've already said the issue can be addressed by adding iDRAC to the list of removable vendors and then it is necessary to rebuild bootstrap initramfs with the fixed version of agent.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/106309

Changed in fuel:
assignee: Miroslav Anashkin (manashkin) → Vladimir Kozhukalov (kozhukalov)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/106312

Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

cd /root
git clone https://github.com/stackforge/fuel-web.git
cd fuel-web
git fetch https://review.openstack.org/stackforge/fuel-web refs/changes/09/106309/1 && git checkout FETCH_HEAD

cd /var/www/nailgun/bootstrap
mkdir new_initramfs
cd new_initramfs
gunzip -c ../initramfs.img | cpio -dvi
cp /root/fuel-web/bin/agent /opt/nailgun/bin
find | grep -v "^\.$" | cpio -H newc -o | gzip -c > ../new_initramfs.img
cd ../
mv initramfs.img initramfs.img.bak
mv new_initramfs.img initramfs.img
cobbler sync

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/5.1.x
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Miroslav, could you describe root cause of this issue and steps for solving it?

Changed in fuel:
assignee: Vladimir Kozhukalov (kozhukalov) → Miroslav Anashkin (manashkin)
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/5.0)

Change abandoned by Dmitry Pyzhov (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/106312

Revision history for this message
Miroslav Anashkin (manashkin) wrote :

This bug appeared against a customozed 5.0 ISO, with (guessing) improperly configured RAID array on the server side and with an active hardware firewall inside the Fuel Admin network.
It has been fixed before I had a look to this issue.
So, bug is invalid.

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Yuri Shovkoplias (yuri-shovkoplias) wrote :

Gentlemen, I am a bit confused.

Do we understand the root cause of the issue and if we do (which I now doubt), how can we reliably resolve it for our customer (Tata Communications)?
This must be addressed before we consider this deployment completed (we have 3 weeks for everything: deployment, testing, etc.).

1. In the earlier comment you have suggested that the issue can be addressed by adding iDRAC to the list of removable vendors and then it is necessary to rebuild bootstrap initramfs with the fixed version of agent (which solves the issue).
2. Then you have implemented the resolution to stable/5.0 branch.
3. Then the resolution was abandoned.
4. Then it was stated (and Miroslav was obviously not sure in this), that there is improperly configured RAID array on the server side and with an active hardware firewall inside the Fuel Admin network.

From a customer perspective this all sounds a bit strange.

Can we get somebody from product group, work with Anton Tarasov, do the root cause analysis and implement the resolution?
We will then rebuild the ISO and will do the cloud redeployment.

Regards,
Yuri

Changed in fuel:
status: Invalid → New
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Raising this bug to High, as it:
1) customer-found defect
2) makes it unable to deploy OpenStack at all

We need a lab to reproduce the issue on it.

Changed in fuel:
importance: Medium → High
Mike Scherbakov (mihgen)
Changed in fuel:
assignee: Miroslav Anashkin (manashkin) → Vladimir Kozhukalov (kozhukalov)
Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

Yuri, Tony I'll be glad to help with this case. Last time when I proposed to add iDRAC into the list of removable devices I certainly made a mistake because it was kind of built-in USB disk. Actually, Tony then re-built bootstrap initramfs (with iDRAC removable fix) and then reported that the problem was not actual. But maybe it was because for his case that day it was enough to have USB disk (I am not sure).

To make this clear we need to have an access to the lab where this server is installed. I'd like to know exactly how hardware RAID and other hardware pieces are configured. I'd like to know exactly which disks are exposed into sysfs, which of them are removable, which links and device files udev creates for those disks.

Revision history for this message
Yuri Shovkoplias (yuri-shovkoplias) wrote :

As discussed in a different thread, Vladimir will look more on this issue.

Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

Yesterday Tony checked if vanilla bootstrap image from 5.0 iso is able to detect appropriate hard drives on a node and it turned out that it is able. So, the problem was not connected to driver compatibility lack. Most probably the reason why this issue appeared is that raid had been configured incorrectly.

Bug is not actual.

Changed in fuel:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Vladimir Kozhukalov (<email address hidden>) on branch: master
Review: https://review.openstack.org/106309

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.