ZFSonLinux CI job falls into endless loop trying to upgrade grub

[Valeriy Ponomaryov]

ZFSonLinux CI job started failing with grub error performing "apt-get update" operation.
It falls into endless loop of picking up proper storage device.

WARNING! Huge(!) log: http://logs.openstack.org/47/350647/6/check/gate-manila-tempest-dsvm-postgres-zfsonlinux/582c093/logs/devstacklog.txt.gz

Paste: http://paste.openstack.org/show/547902/

Raw:

2016-08-04 07:13:56.343 | ++ /opt/stack/new/manila/devstack/plugin.sh:install_manila:581 : sudo apt-get -y -q upgrade
2016-08-04 07:13:56.353 | Reading package lists...
2016-08-04 07:13:56.488 | Building dependency tree...
2016-08-04 07:13:56.489 | Reading state information...
2016-08-04 07:13:56.632 | The following packages will be upgraded:
2016-08-04 07:13:56.632 | base-files dkms grub-common grub-pc grub-pc-bin grub2-common initramfs-tools
2016-08-04 07:13:56.632 | initramfs-tools-bin isc-dhcp-client isc-dhcp-common libdrm2 libxrandr2
2016-08-04 07:13:56.632 | mountall
2016-08-04 07:13:56.641 | 13 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2016-08-04 07:13:56.641 | Need to get 4883 kB of archives.
2016-08-04 07:13:56.641 | After this operation, 7168 B of additional disk space will be used.
2016-08-04 07:13:56.641 | WARNING: The following packages cannot be authenticated!
2016-08-04 07:13:56.641 | base-files libdrm2 initramfs-tools initramfs-tools-bin libxrandr2
2016-08-04 07:13:56.641 | isc-dhcp-client isc-dhcp-common dkms grub-pc grub-pc-bin grub2-common
2016-08-04 07:13:56.641 | grub-common
2016-08-04 07:13:56.650 | Authentication warning overridden.
2016-08-04 07:13:56.650 | Get:1 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main base-files amd64 7.2ubuntu5.5 [67.5 kB]
2016-08-04 07:13:56.653 | Get:2 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main libdrm2 amd64 2.4.67-1ubuntu0.14.04.1 [27.1 kB]
2016-08-04 07:13:56.656 | Get:3 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main initramfs-tools all 0.103ubuntu4.4 [44.5 kB]
2016-08-04 07:13:56.658 | Get:4 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main initramfs-tools-bin amd64 0.103ubuntu4.4 [9134 B]
2016-08-04 07:13:56.660 | Get:5 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main libxrandr2 amd64 2:1.5.0-1~trusty1 [17.5 kB]
2016-08-04 07:13:56.663 | Get:6 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main isc-dhcp-client amd64 4.2.4-7ubuntu12.5 [639 kB]
2016-08-04 07:13:56.676 | Get:7 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main isc-dhcp-common amd64 4.2.4-7ubuntu12.5 [707 kB]
2016-08-04 07:13:56.690 | Get:8 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main dkms all 2.2.0.3-1.1ubuntu5.14.04.7 [66.1 kB]
2016-08-04 07:13:56.693 | Get:9 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main grub-pc amd64 2.02~beta2-9ubuntu1.12 [173 kB]
2016-08-04 07:13:56.698 | Get:10 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main grub-pc-bin amd64 2.02~beta2-9ubuntu1.12 [880 kB]
2016-08-04 07:13:56.716 | Get:11 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main grub2-common amd64 2.02~beta2-9ubuntu1.12 [501 kB]
2016-08-04 07:13:56.726 | Get:12 http://mirror.regionone.osic-cloud1.openstack.org/ubuntu/ trusty-updates/main grub-common amd64 2.02~beta2-9ubuntu1.12 [1681 kB]
2016-08-04 07:13:56.914 | Get:13 http://ppa.launchpad.net/zfs-native/stable/ubuntu/ trusty/main mountall amd64 2.53-zfs1 [69.9 kB]
2016-08-04 07:13:57.335 | debconf: unable to initialize frontend: Dialog
2016-08-04 07:13:57.335 | debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2016-08-04 07:13:57.335 | debconf: falling back to frontend: Readline
2016-08-04 07:13:57.341 | debconf: unable to initialize frontend: Readline
2016-08-04 07:13:57.342 | debconf: (This frontend requires a controlling tty.)
2016-08-04 07:13:57.342 | debconf: falling back to frontend: Teletype
2016-08-04 07:13:57.345 | dpkg-preconfigure: unable to re-open stdin:
2016-08-04 07:13:57.355 | Fetched 4883 kB in 0s (7933 kB/s)
2016-08-04 07:13:57.413 | (Reading database ... 78333 files and directories currently installed.)
2016-08-04 07:13:57.414 | Preparing to unpack .../base-files_7.2ubuntu5.5_amd64.deb ...
2016-08-04 07:13:57.430 | Unpacking base-files (7.2ubuntu5.5) over (7.2ubuntu5.4) ...
2016-08-04 07:13:57.494 | Setting up base-files (7.2ubuntu5.5) ...
2016-08-04 07:13:57.504 | Installing new version of config file /etc/issue ...
2016-08-04 07:13:57.506 | Installing new version of config file /etc/issue.net ...
2016-08-04 07:13:57.509 | Installing new version of config file /etc/lsb-release ...
2016-08-04 07:13:57.510 | Installing new version of config file /etc/os-release ...
2016-08-04 07:13:57.591 | (Reading database ... 78333 files and directories currently installed.)
2016-08-04 07:13:57.592 | Preparing to unpack .../libdrm2_2.4.67-1ubuntu0.14.04.1_amd64.deb ...
2016-08-04 07:13:57.597 | Unpacking libdrm2:amd64 (2.4.67-1ubuntu0.14.04.1) over (2.4.64-1~ubuntu14.04.1) ...
2016-08-04 07:13:57.654 | Preparing to unpack .../initramfs-tools_0.103ubuntu4.4_all.deb ...
2016-08-04 07:13:57.660 | Unpacking initramfs-tools (0.103ubuntu4.4) over (0.103ubuntu4.3) ...
2016-08-04 07:13:57.703 | Preparing to unpack .../initramfs-tools-bin_0.103ubuntu4.4_amd64.deb ...
2016-08-04 07:13:57.708 | Unpacking initramfs-tools-bin (0.103ubuntu4.4) over (0.103ubuntu4.3) ...
2016-08-04 07:13:57.731 | Preparing to unpack .../mountall_2.53-zfs1_amd64.deb ...
2016-08-04 07:13:57.736 | Unpacking mountall (2.53-zfs1) over (2.53) ...
2016-08-04 07:13:57.794 | Preparing to unpack .../libxrandr2_2%3a1.5.0-1~trusty1_amd64.deb ...
2016-08-04 07:13:57.799 | Unpacking libxrandr2:amd64 (2:1.5.0-1~trusty1) over (2:1.4.2-1) ...
2016-08-04 07:13:57.828 | Preparing to unpack .../isc-dhcp-client_4.2.4-7ubuntu12.5_amd64.deb ...
2016-08-04 07:13:57.838 | Unpacking isc-dhcp-client (4.2.4-7ubuntu12.5) over (4.2.4-7ubuntu12.4) ...
2016-08-04 07:13:57.929 | Preparing to unpack .../isc-dhcp-common_4.2.4-7ubuntu12.5_amd64.deb ...
2016-08-04 07:13:57.934 | Unpacking isc-dhcp-common (4.2.4-7ubuntu12.5) over (4.2.4-7ubuntu12.4) ...
2016-08-04 07:13:58.020 | Preparing to unpack .../dkms_2.2.0.3-1.1ubuntu5.14.04.7_all.deb ...
2016-08-04 07:13:58.026 | Unpacking dkms (2.2.0.3-1.1ubuntu5.14.04.7) over (2.2.0.3-1.1ubuntu5.14.04.6) ...
2016-08-04 07:13:58.110 | Preparing to unpack .../grub-pc_2.02~beta2-9ubuntu1.12_amd64.deb ...
2016-08-04 07:13:58.119 | Unpacking grub-pc (2.02~beta2-9ubuntu1.12) over (2.02~beta2-9ubuntu1.11) ...
2016-08-04 07:13:58.149 | Preparing to unpack .../grub-pc-bin_2.02~beta2-9ubuntu1.12_amd64.deb ...
2016-08-04 07:13:58.153 | Unpacking grub-pc-bin (2.02~beta2-9ubuntu1.12) over (2.02~beta2-9ubuntu1.11) ...
2016-08-04 07:13:58.306 | Preparing to unpack .../grub2-common_2.02~beta2-9ubuntu1.12_amd64.deb ...
2016-08-04 07:13:58.311 | Unpacking grub2-common (2.02~beta2-9ubuntu1.12) over (2.02~beta2-9ubuntu1.11) ...
2016-08-04 07:13:58.391 | Preparing to unpack .../grub-common_2.02~beta2-9ubuntu1.12_amd64.deb ...
2016-08-04 07:13:58.426 | Unpacking grub-common (2.02~beta2-9ubuntu1.12) over (2.02~beta2-9ubuntu1.11) ...
2016-08-04 07:13:58.686 | Setting up libdrm2:amd64 (2.4.67-1ubuntu0.14.04.1) ...
2016-08-04 07:13:58.697 | Setting up initramfs-tools-bin (0.103ubuntu4.4) ...
2016-08-04 07:13:58.701 | Setting up initramfs-tools (0.103ubuntu4.4) ...
2016-08-04 07:13:58.717 | update-initramfs: deferring update (trigger activated)
2016-08-04 07:13:58.720 | Setting up mountall (2.53-zfs1) ...
2016-08-04 07:13:58.767 | Setting up libxrandr2:amd64 (2:1.5.0-1~trusty1) ...
2016-08-04 07:13:58.777 | Setting up isc-dhcp-common (4.2.4-7ubuntu12.5) ...
2016-08-04 07:13:58.781 | Setting up isc-dhcp-client (4.2.4-7ubuntu12.5) ...
2016-08-04 07:13:58.783 | Installing new version of config file /etc/apparmor.d/sbin.dhclient ...
2016-08-04 07:13:58.855 | debconf: unable to initialize frontend: Dialog
2016-08-04 07:13:58.855 | debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2016-08-04 07:13:58.855 | debconf: falling back to frontend: Readline
2016-08-04 07:13:58.860 | debconf: unable to initialize frontend: Readline
2016-08-04 07:13:58.860 | debconf: (This frontend requires a controlling tty.)
2016-08-04 07:13:58.861 | debconf: falling back to frontend: Teletype
2016-08-04 07:13:58.870 | Setting up dkms (2.2.0.3-1.1ubuntu5.14.04.7) ...
2016-08-04 07:13:58.917 | Setting up grub-common (2.02~beta2-9ubuntu1.12) ...
2016-08-04 07:13:58.937 | Installing new version of config file /etc/grub.d/30_uefi-firmware ...
2016-08-04 07:13:58.990 | Setting up grub2-common (2.02~beta2-9ubuntu1.12) ...
2016-08-04 07:13:58.995 | Setting up grub-pc-bin (2.02~beta2-9ubuntu1.12) ...
2016-08-04 07:13:59.012 | Setting up grub-pc (2.02~beta2-9ubuntu1.12) ...
2016-08-04 07:13:59.097 | debconf: unable to initialize frontend: Dialog
2016-08-04 07:13:59.097 | debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2016-08-04 07:13:59.097 | debconf: falling back to frontend: Readline
2016-08-04 07:13:59.104 | debconf: unable to initialize frontend: Readline
2016-08-04 07:13:59.104 | debconf: (This frontend requires a controlling tty.)
2016-08-04 07:13:59.104 | debconf: falling back to frontend: Teletype
2016-08-04 07:13:59.289 | length() used on @choices (did you mean "scalar(@choices)"?) at /usr/share/perl5/Debconf/Element/Teletype/Select.pm line 48, <GEN6> line 10.
2016-08-04 07:13:59.289 | length() used on @choices (did you mean "scalar(@choices)"?) at /usr/share/perl5/Debconf/Element/Teletype/Select.pm line 63, <GEN6> line 10.
2016-08-04 07:13:59.290 | Configuring grub-pc
2016-08-04 07:13:59.290 | -------------------
2016-08-04 07:13:59.290 |
2016-08-04 07:13:59.290 | A new version of configuration file /etc/default/grub is available, but the
2016-08-04 07:13:59.290 | version installed currently has been locally modified.
2016-08-04 07:13:59.290 |
2016-08-04 07:13:59.290 | 1. install the package maintainer's version
2016-08-04 07:13:59.290 | 2. keep the local version currently installed
2016-08-04 07:13:59.290 | 3. show the differences between the versions
2016-08-04 07:13:59.290 | 4. show a side-by-side difference between the versions
2016-08-04 07:13:59.290 | 5. show a 3-way difference between available versions
2016-08-04 07:13:59.290 | 6. do a 3-way merge between available versions (experimental)
2016-08-04 07:13:59.290 | 7. start a new shell to examine the situation
2016-08-04 07:13:59.290 | What do you want to do about modified configuration file grub?
2016-08-04 07:13:59.290 | Use of uninitialized value $_[1] in join or string at /usr/share/perl5/Debconf/DbDriver/Stack.pm line 111.
2016-08-04 07:13:59.443 | The grub-pc package is being upgraded. This menu allows you to select which
2016-08-04 07:13:59.443 | devices you'd like grub-install to be automatically run for, if any.
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | Running grub-install automatically is recommended in most situations, to
2016-08-04 07:13:59.443 | prevent the installed GRUB core image from getting out of sync with GRUB
2016-08-04 07:13:59.443 | modules or grub.cfg.
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | If you're unsure which drive is designated as boot drive by your BIOS, it is
2016-08-04 07:13:59.443 | often a good idea to install GRUB to all of them.
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | Note: it is possible to install GRUB to partition boot records as well, and
2016-08-04 07:13:59.443 | some appropriate partitions are offered here. However, this forces GRUB to use
2016-08-04 07:13:59.443 | the blocklist mechanism, which makes it less reliable, and therefore is not
2016-08-04 07:13:59.443 | recommended.
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | 1. /dev/vda (103079 MB; ???) 3. /dev/vda1 (103078 MB; ???)
2016-08-04 07:13:59.443 | 2. /dev/vdb (2147 MB; ???)
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | (Enter the items you want to select, separated by spaces.)
2016-08-04 07:13:59.443 |
2016-08-04 07:13:59.443 | GRUB install devices:
2016-08-04 07:13:59.443 | Use of uninitialized value $_[1] in join or string at /usr/share/perl5/Debconf/DbDriver/Stack.pm line 111.
2016-08-04 07:13:59.451 | You chose not to install GRUB to any devices. If you continue, the boot loader
2016-08-04 07:13:59.451 | may not be properly configured, and when this computer next starts up it will
2016-08-04 07:13:59.451 | use whatever was previously in the boot sector. If there is an earlier version
2016-08-04 07:13:59.451 | of GRUB 2 in the boot sector, it may be unable to load modules or handle the
2016-08-04 07:13:59.451 | current configuration file.
2016-08-04 07:13:59.451 |
2016-08-04 07:13:59.451 | If you are already using a different boot loader and want to carry on doing so,
2016-08-04 07:13:59.451 | or if this is a special environment where you do not need a boot loader, then
2016-08-04 07:13:59.451 | you should continue anyway. Otherwise, you should install GRUB somewhere.
2016-08-04 07:13:59.451 |
2016-08-04 07:13:59.451 | Continue without installing GRUB?
2016-08-04 07:13:59.451 | Use of uninitialized value $_[1] in join or string at /usr/share/perl5/Debconf/DbDriver/Stack.pm line 111.
2016-08-04 07:13:59.535 | The grub-pc package is being upgraded. This menu allows you to select which
2016-08-04 07:13:59.535 | devices you'd like grub-install to be automatically run for, if any.
2016-08-04 07:13:59.535 |
2016-08-04 07:13:59.535 | Running grub-install automatically is recommended in most situations, to
2016-08-04 07:13:59.535 | prevent the installed GRUB core image from getting out of sync with GRUB
2016-08-04 07:13:59.535 | modules or grub.cfg.
2016-08-04 07:13:59.535 |
2016-08-04 07:13:59.535 | If you're unsure which drive is designated as boot drive by your BIOS, it is
2016-08-04 07:13:59.535 | often a good idea to install GRUB to all of them.
2016-08-04 07:13:59.535 |
2016-08-04 07:13:59.535 | Note: it is possible to install GRUB to partition boot records as well, and
2016-08-04 07:13:59.535 | some appropriate partitions are offered here. However, this forces GRUB to use
2016-08-04 07:13:59.535 | the blocklist mechanism, which makes it less reliable, and therefore is not
2016-08-04 07:13:59.535 | recommended.
2016-08-04 07:13:59.535 |
2016-08-04 07:13:59.535 | 1. /dev/vda (103079 MB; ???) 3. /dev/vda1 (103078 MB; ???)
2016-08-04 07:13:59.535 | 2. /dev/vdb (2147 MB; ???)
2016-08-04 07:13:59.535 |
2016-08-04 07:13:59.535 | (Enter the items you want to select, separated by spaces.)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351110

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351110
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=88fb70f92bee65299691daecf4ccb9cfa2e66f26
Submitter: Jenkins
Branch: master

commit 88fb70f92bee65299691daecf4ccb9cfa2e66f26
Author: Valeriy Ponomaryov <email address hidden>
Date: Thu Aug 4 13:24:30 2016 +0300

    Fix ZFSonLinux driver prerequisites setup

    We call "apt-get upgrade" to apply changes related to ppa
    of ZFS on linux. But "grub"-related packaged started making
    dialog that hangs forever in case of automatic setup.
    So, exclude "grub" related packages from update as redundant
    for unblocking ZFSonLinux driver prerequisites setup.

    Change-Id: Ibb94604710b0d4199825e770194655653c26bf89
    Closes-Bug: #1609696

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 3.0.0.0b3 development milestone.