grub-install takes too long when a number of disks is high

[Alexander Gordeev]

Detailed bug description:
 apparently, fuel-agent installs grub stage0 on all of disks nailgun knows of.

 https://github.com/openstack/fuel-agent/blob/master/fuel_agent/drivers/nailgun.py#L340-L341

 it's done intentionally, this step allows us to boot from any of those disks. Without that, operator/administrator have to set proper boot order for disks in BIOS/UEFI setup utility for every single node.

 it looks like grub-install spends a lot of time with analyzing all of disks with os-proper scripts. OS-prober scripts try to find presence of any other OS (operating system) to add it then to grub menu, thus allowing the user to choose which OS to boot. It seems it's not a case for Fuel as grub.conf file will be configured by fuel-agent https://github.com/openstack/fuel-agent/blob/master/fuel_agent/utils/grub.py#L230-L254

 So, we're dealing with O(n^2) time complexity as every single call of grub-install will cause OS-prober to analyze all disks unconditionally.

 I propose to tune OS-proper scripts or just disable them completely (Not sure which way is more better and easier to achieve). As fuel-agent controls everything explicitly in a data driven way. We're not in a need of additional logic from GRUB. Fuel-agent better knows where to install and how to configure GRUB bootloader for Fuel.

Steps to reproduce:
  provision a node with huge number of disks. 20 or 30 may be good enough. Last stage of provisioning could take few minutes.
Expected results:
 provisioning time won't be tied directly to a number of disks.
Actual result:
 provisioning time is increasing. The latest step of provisioning, which usually lasts just a dozen of seconds, is almost squared.
Reproducibility:
 every single time on a node with huge number of disks
Workaround:
 unknown
Impact:
 the last step of provisioning could take more amount of time than the provisioning itself. Horrible UX.

version: 9.0 (6.1+)

[Aleksey Zvyagintsev (azvyagintsev)]

CHeck timestamps , on system with 64 drives:

2016-03-18 13:39:25.030 8862 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/mapper/3600144f0534f392c000056e972830002
....
2016-03-18 13:47:41.209 8862 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/sda

[Alexander Gordeev (a-gordeev)]

https://review.openstack.org/#/c/285453/6/bareon/utils/grub.py@269 could it be resolved by this just one line?

[Aleksey Zvyagintsev (azvyagintsev)]

2016-03-21 11:30:36.994 31729 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/mapper/3600144f0534f392c000056e972830002 --skip-fs-probe
2016-03-21 11:30:44.984 31729 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/mapper/3600144f0534f392c000056e972830003 --skip-fs-probe

...
2016-03-21 11:39:00.551 31729 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/mapper/3600144f0534f392c000056e9728b0041 --skip-fs-probe
2016-03-21 11:39:08.471 31729 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install /dev/sda --skip-fs-probe

Summary - --skip-fs-probe didn't help. moving bug back to python team.

[Aleksey Zvyagintsev (azvyagintsev)]

2016-03-21T14:23:01.184944+00:00 info: 2016-03-21 14:23:01.090 15687 DEBUG fuel_agent.utils.utils [-] Env variables: {'TERM': 'linux', 'GRUB_DISABLE_OS_PROBER': 'true', 'PWD': '/', 'UPSTART_INSTANCE': '', 'UPSTART_JOB': 'mcollective', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin', 'LC_ALL': 'C'}
2016-03-21T14:23:08.197935+00:00 info: 2016-03-21 14:23:08.144 15687 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install --skip-fs-probe /dev/mapper/3600144f0534f392c000056e972830003
2016-03-21T14:23:08.197935+00:00 info: 2016-03-21 14:23:08.145 15687 DEBUG fuel_agent.utils.utils [-] Env variables: {'LANG': 'C', 'TERM': 'linux', 'GRUB_DISABLE_OS_PROBER': 'true', 'LANGUAGE': 'C', 'PWD': '/', 'UPSTART_INSTANCE': '', 'UPSTART_JOB': 'mcollective', 'PATH': '/bin:/usr/bin:/sbin:/usr/sbin', 'LC_ALL': 'C'}
.....
2016-03-21T14:31:35.152709+00:00 info: 2016-03-21 14:31:34.837 15687 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install --skip-fs-probe /dev/mapper/3600144f0534f392c000056e9728b0041
2016-03-21T14:31:35.153341+00:00 info: 2016-03-21 14:31:34.838 15687 DEBUG fuel_agent.utils.utils [-] Env variables: {'LANG': 'C', 'TERM': 'linux', 'GRUB_DISABLE_OS_PROBER': 'true', 'LANGUAGE': 'C', 'PWD': '/', 'UPSTART_INSTANCE': '', 'UPSTART_JOB': 'mcollective', 'PATH': '/bin:/usr/bin:/sbin:/usr/sbin', 'LC_ALL': 'C'}
2016-03-21T14:31:42.166495+00:00 info: 2016-03-21 14:31:41.904 15687 DEBUG fuel_agent.utils.utils [-] Trying to execute command: chroot /tmp/target /usr/sbin/grub-install --skip-fs-probe /dev/sda
2016-03-21T14:31:42.166495+00:00 info: 2016-03-21 14:31:41.905 15687 DEBUG fuel_agent.utils.utils [-] Env variables: {'LANG': 'C', 'TERM': 'linux', 'GRUB_DISABLE_OS_PROBER': 'true', 'LANGUAGE': 'C', 'PWD': '/', 'UPSTART_INSTANCE': '', 'UPSTART_JOB': 'mcollective', 'PATH': '/bin:/usr/bin:/sbin:/usr/sbin', 'LC_ALL': 'C'}

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Gordeev (a-gordeev)]

Moved to 10.0 due to low impact. Doesn't break real deployment. It just delays provisioning for some predictable amount of time.

So, for large number of disks it could lead to take additional few minutes more.