PowerVM: ubuntu-14.04 stuck in ibm,client-architecture-support reboot loop

[bugproxy]

---Problem Description---
This bug is a follow-up of Bug#110009. The problem here is after successful installation of Ubuntu 14.04, the system fails to boot the OS. When I select installed OS "Ubuntu" from grub menu, it throws below error and falls back to GRUB menu again and again:

..........................................................................
..........................................................................
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.13.0-24-generic (buildd@fisher04) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #46-Ubuntu SMP Thu Apr 10 19:09:21 UTC 2014 (Ubuntu 3.13.0-24.46-generic 3.13.9)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 256 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...
ibm,sp /vdevice/IBM,sp@4000
\
ibm,sp /vdevice/IBM,sp@4000
\
Elapsed time since release of system processors: 10675 mins 3 secs
error: no suitable video mode found.

Machine Type = IBM,8286-42A Mach serial number: 1069B3T

---Steps to Reproduce---
1) Try virtual DVD installation
2) Boot the LPAR via cdrom
    i.e try below commands from openfirmware prompt:

    0> devalias cdrom /vdevice/v-scsi@30000010/disk@8300000000000000
    0> boot cdrom

3) proceed with default installation, 100% installation completes. Boot the installed OS from hard disk and you hit this "no suitable video mode found" error and system falls back to GRUB.

Install method: virtual DVD

Install ISO Information: ubuntu-14.04-server-ppc64el.iso

*Additional Instructions for backup: <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.

== Comment: #1 - David Heller <email address hidden> - 2014-06-11 15:32:48 ==
This seems to be a pretty common problem, albeit with multiple causes. I see that you already tried some of the workarounds in similar bug here:

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/699802

Still, it seems likely this is due to the video mode that grub is attempting to user for the splash screen. Either that, or it's a video driver issue at the grub level.

Can you please try the procedure in comment #4 here:

http://wwww.ubuntuforums.org/showthread.php?t=1471399

This involves booting to an alternate method, such as a live cd, and operating on the grub config files in the newly installed system. If you are unable to, or have no success with updating the files, it would at least be good to collect the files and attach the information here, before sending this back to launchpad

If you are successful in booting to live CD on the same system where the boot fails, it would be good to compare the files from live CD to those in the failing system.

Also, do you mind installing with Ubuntu 14.10 alpha images?

http://cdimage.ubuntu.com/ubuntu-server/daily/current/utopic-server-ppc64el.iso

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-06-26 17:44 EDT-------
The message "error: no suitable video mode found." is not really an error, it's just GRUB saying that it didn't found any display card and will run in text mode. I almost sure that this is not the cause.

For me, the real culprit is the "IBM client architecture reboot" support. Here, the last message before the reboot is "Calling ibm,client-architecture-support...".

Linux kernel calls the method and for some reason it keeps rebooting over and over.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1334793/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-06-27 11:45 EDT-------
(In reply to comment #10)
> The message "error: no suitable video mode found." is not really an error,
> it's just GRUB saying that it didn't found any display card and will run in
> text mode. I almost sure that this is not the cause.
>
> For me, the real culprit is the "IBM client architecture reboot" support.
> Here, the last message before the reboot is "Calling
> ibm,client-architecture-support...".
>
> Linux kernel calls the method and for some reason it keeps rebooting over
> and over.

Well, I tried the procedure mentioned in comment#4 of http://wwww.ubuntuforums.org/showthread.php?t=1471399, by booting my system into rescue mode. But this did not help me either, still same behaviour!

Paulo could be right, the main culprit seems to be at this point: "Calling ibm,client-architecture-support...\"
Usually I have seen output of above command either as "done" or "not implemented", but in this case it is just ...(nil).

Is open firmware failing to validate for "client-architecture-support" here? OR something missing in pHyp support?

------- Comment From <email address hidden> 2014-06-27 11:46 EDT-------
(In reply to comment #6)
> Also, do you mind installing with Ubuntu 14.10 alpha images?
>
> http://cdimage.ubuntu.com/ubuntu-server/daily/current/utopic-server-ppc64el.
> iso

Sure, I will give this try on Monday.

Thanks!

------- Comment From <email address hidden> 2014-07-01 06:18 EDT-------
(In reply to comment #14)
> (In reply to comment #6)
> > Also, do you mind installing with Ubuntu 14.10 alpha images?
> >
> > http://cdimage.ubuntu.com/ubuntu-server/daily/current/utopic-server-ppc64el.
> > iso
>
> Sure, I will give this try on Monday.
>
> Thanks!

Well..this issue is reproducible with Ubuntu14.10 alpha as well.

FYI, here is the console log:

GNU GRUB version 2.02~beta2-10

+----------------------------------------------------------------------------+
|*Ubuntu |
| Advanced options for Ubuntu |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+----------------------------------------------------------------------------+

Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.

OF stdout device is: /vdevice/vty@300...

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1334793

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-08-27 15:01 EDT-------
*** Bug 115197 has been marked as a duplicate of this bug. ***

[Ivan Warren (ivmn)]

I can confirm this issue.

I have a S814 system running PowerVM and although the install completed, I boot loop on the "Calling ibm,client-architecture-support..." message.

NOTE : I installed debian powerpc for a test - and also got this message - and the partition rebooted - but only once - then all proceeded as expected.

--Ivan

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-03 22:43 EDT-------
(In reply to comment #43)
> I can confirm this issue.
>
> I have a S814 system running PowerVM and although the install completed, I
> boot loop on the "Calling ibm,client-architecture-support..." message.
>
> NOTE : I installed debian powerpc for a test - and also got this message -
> and the partition rebooted - but only once - then all proceeded as expected.

Just as an FYI, I was able to workaround the issue (temporarily until we find the fix) by using an internal version of grub (used for NIM installs). I netbooted my LPAR that was failing as above and used the on-disk conf file (well, copied it to my boot-server and used it as the netboot conf file). The system rebooted once for c-a-s negotiation, and then successfully booted to login prompt!

So, I believe the issue is in the grub shipped with Ubuntu. I did a quick test of setting an nvram variable at the OF prompt (ibm,fw-new-mem-def) to true, and then booting to grub, rebooting from the grub prompt and checking the variable again. It's value was now false. The Linux kernel negotiates this value to true, so I think that's why we end up in a c-a-s loop. Assigning over to Paulo for now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02~beta2-14

---------------
grub2 (2.02~beta2-14) unstable; urgency=medium

  * On architectures without a real GRUB port, just build the utilities.
    This makes tools such as grub-probe and grub-fstest available
    everywhere, and makes grub-mount available on all Linux and kFreeBSD
    architectures.
  * Remove .MIPS.abiflags section from images (thanks, Jurica Stanojkovic,
    although I used a slightly simpler approach; closes: #762307).
  * Include a text attribute reset in the clear command for ppc
    (LP: #1295255).
  * Disable VSX instruction on powerpc startup to fix booting on ppc64el.
  * Stop adding a CHRP note on chrp_ibm machines, since that apparently
    breaks PowerVM and isn't needed on other machine types as far as we can
    tell (LP: #1334793).
  * Refactor flicker-free-boot configuration in debian/rules to reduce
    duplication.
  * Disable flicker-free-boot on Ubuntu ppc64el for now, as it isn't needed
    there and causes too many problems (LP: #1338471).
  * Use nvram rather than nvsetenv on chrp_ibm machines, since that tool is
    better-supported and copes with such things as nvram being missing in
    qemu.
  * Remove brace-expansion from the postrm, and switch the preinst and
    postrm to /bin/sh (closes: #762940).
  * On ppc64el, look for a PReP partition and install the core image to the
    first one if found. For now this is done by borrowing prep-bootdev.c
    from grub-installer, incurring a dependency on libparted.

 -- Colin Watson <email address hidden> Fri, 26 Sep 2014 18:05:15 +0100

[Breno Leitão (breno-leitao)]

I understand that this bug makes 14.10 only, but not 14.04, right?

14.04.2 is going to officially support PowerVM hypervisor and this fix is required to do this work.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-03-03 14:25 EDT-------
*** Bug 121986 has been marked as a duplicate of this bug. ***

[Chris J Arges (arges)]

Hello bugproxy, or anyone else affected,

Accepted grub2 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-9ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Paulo Flabiano Smorigo (pfsmorigo)]

Just installed in a PowerVM and updated the grub2 package to 2.02~beta2-9ubuntu1.2. It worked fine.

[Chris J Arges (arges)]

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2-signed - 1.34.3

---------------
grub2-signed (1.34.3) trusty; urgency=medium

  * Rebuild against grub-efi-amd64 2.02~beta2-9ubuntu1.2 (LP: #1334793)

 -- Adam Conrad <email address hidden> Mon, 18 May 2015 03:07:25 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02~beta2-9ubuntu1.2

---------------
grub2 (2.02~beta2-9ubuntu1.2) trusty-proposed; urgency=medium

  * debian/patches/install_powerpc_machtypes.patch: updated: do a better job
    at detecting machine types; so as to use the right utility when updating
    nvram for the boot-device. This also fixes adding a CHRP note on the
    chrp_ibm machines, which broke PowerVM mode. (LP: #1334793)
  * debian/patches/ppc64el-disable-vsx.patch: disable the VSX instruction,
    which is enabled by default on POWER7/8 cpu models, to avoid crashes due
    to instruction exceptions. The kernel will re-enable it when necessary.
    (LP: #1454743)
  * debian/patches/ieee1275-clear-reset.patch: clear the text attribute in
    the clear command. (LP: #1454764)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 13 May 2015 12:30:05 -0400