Ubuntu 15.10 daily ISO installation fails to reboot with Program Exception

Bug #1459706 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Undecided
Unassigned

Bug Description

== Comment: #0 - SACHIN P. SANT <email address hidden> - 2015-05-27 02:05:10 ==
---Problem Description---
Ubuntu 15.10 daily ISO installation fails to reboot with Program Exception

Contact Information = Sachin Sant / <email address hidden>

---uname output---
3.19.0-18-generic

Machine Type = POWER8

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1) Start Ubuntu 15.10 installation as a Power KVM guest.
2) Post installation guest will be rebooted.
3) Post reboot guest fails with following messages :

Escape character is ^]

SLOF **********************************************************************
QEMU Starting
 Build Date = Mar 17 2015 15:44:58
 FW Version = mockbuild@ release 20150317
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/l-lan@1000
Populating /vdevice/v-scsi@2000
       SCSI: Looking for devices
          8001000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.0."
          8000000000000000 DISK : "QEMU QEMU HARDDISK 2.0."
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
 Adapters on 0800000020000000
                     00 0800 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 1000 (D) : 1af4 1002 unknown-legacy-device*
No NVRAM common partition, re-initializing...
Scanning USB
  OHCI: initializing
Using default console: /vdevice/vty@30000000

  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /vdevice/v-scsi@2000/disk@8000000000000000 ...
No DOS disk-label found.
  Successfully loaded

( 700 ) Program Exception [ 0 ]

    R0 .. R7 R8 .. R15 R16 .. R23 R24 .. R31
0000000000000000 0000000000000000 0000000000000000 000000007dd12ad0
000000007dd12ac0 0000000061676963 0000000000000000 000000000021aad8
0000000000000000 0000000000000000 0000000000657269 00000000c0000000
0000000000181210 000000007dd12ac0 000000006300656e 0000000055654f44
00000000001812a4 0000000000200360 0000000065726963 0000000000000000
0000000000000000 0000000000000000 0000000000d12c28 0000000000000000
0000000000000000 0000000000000000 00000000001ebcf0 000000000021aa1c
000000000020f628 0000000000000000 0000000000000002 0000000000000000

    CR / XER LR / CTR SRR0 / SRR1 DAR / DSISR
        80023022 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000200360 8000000000081000 00000000

3a >

Install method: DVD

Install disk info: d

Install ISO Information: wily-server-ppc64el.iso

*Additional Instructions for Sachin Sant / <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.

== Comment: #2 - Colleen R. Stouffer <email address hidden> - 2015-05-27 17:57:50 ==
This system does not use Partition Firmware. KVM uses SLOF.

The error is a Program Exception 700, which is an illegal instruction. The
address of the instruction in SRR0 is 0. Code is attempting to execute at that
address.

== Comment: #6 - David Heller <email address hidden> - 2015-05-28 10:23:37 ==
From LTC bug 116747 Comment 12 Paulo Flabiano Smorigo 2014-09-29 18:21:13 EDT

That exception is caused by a VSX exception (f40 at SRR0) in the release 13 of grub package. A new package (14) was released friday and fixes that problem.

From LTC bug 116747 Comment 16 Paulo Flabiano Smorigo 2014-09-30 14:33:41 EDT

grub is fixed in the ISO but in order to use this new version another package, debian-installer, needed to be rebuilt.

SO it looks like some patches were not forward-ported to grub-installer in 15.10

FYI: Another dup was LTC bug 116767, which was mirrored to LP 1375216.

Related branches

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-125543 severity-critical targetmilestone-inin1510
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-05-28 15:24 EDT-------
This exception isn't the same as the VSX one. The SRR0 is 0 so I believe that it's something else.

I'm not aware of any patch that needs to be forward-ported to 15.10. All necessary patches are in place.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-05-28 15:31 EDT-------
Did some tests and the daily and the problem started on May 23.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1459706/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-05-29 15:59 EDT-------
*** Bug 125469 has been marked as a duplicate of this bug. ***

Dave Heller (hellerda)
affects: ubuntu → debian-installer (Ubuntu)
Revision history for this message
Steve Langasek (vorlon) wrote :

There is an updated build of grub in the wily daily images, compared with the 15.04 release. The source changes to grub should not be an issue, but there have been toolchain updates between the two builds of grub, which may come into play.

From the ppc64el build log of grub2 2.02~beta2-22ubuntu1:
Toolchain package versions: libc6-dev_2.21-0ubuntu4 make_4.0-8.1 dpkg-dev_1.17.24ubuntu1 gcc-4.9_4.9.2-10ubuntu12 g++-4.9_4.9.2-10ubuntu12 binutils_2.25-5ubuntu6 libstdc++-4.9-dev_4.9.2-10ubuntu12 libstdc++6_4.9.2-10ubuntu12

From the ppc64el build log of grub2 2.02~beta2-23:
Toolchain package versions: libc6-dev_2.21-0ubuntu4 make_4.0-8.1 dpkg-dev_1.17.25ubuntu1 gcc-4.9_4.9.2-17ubuntu1 g++-4.9_4.9.2-17ubuntu1 binutils_2.25-5ubuntu7 libstdc++-4.9-dev_4.9.2-17ubuntu1 libstdc++6_5.1.1-5ubuntu4

Revision history for this message
bugproxy (bugproxy) wrote : PowerKVM 3.1: Guest Ubuntu 15.10: same error hit on install-reboot

------- Comment (attachment only) From <email address hidden> 2015-06-08 16:34 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-06-09 18:04 EDT-------
I downgrade gcc and the exception didn't happened.

Looking the changelog between version 4.9_4.9.2-10ubuntu12 and 4.9_4.9.2-17ubuntu1 I found a ppc related comment:

* Configure with --enable-targets=powerpcle-linux on ppc64el for backports to jessie, trusty, utopic and vivid.

[1] http://changelogs.ubuntu.com/changelogs/pool/main/g/gcc-4.9/gcc-4.9_4.9.2-20ubuntu1/changelog

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-06-11 11:47 EDT-------
The exception occurs in the normal.mod, not in the kernel part of GRUB. I compared the old and new gcc normal.mod versions and found this small change:

14978c14978
< 8a08: 7f e1 07 19 stxvw4x vs63,r1,r0
---
> 8a08: 7f e1 01 ce stvx v31,r1,r0
15512c15512
< 8fc0: 7f e1 06 19 lxvw4x vs63,r1,r0
---
> 8fc0: 7f e1 00 ce lvx v31,r1,r0

Both stxvw4x and lxvw4x are VSX instructions. We already have a patch to avoid crashes with those instructions but for some reason it's not enough. I'm going to add a flag in CFLAGS in order to disable VSX at all.

Revision history for this message
bugproxy (bugproxy) wrote : powerpc: Add a flag to avoid unnecessary optimizations (like vsx)

------- Comment on attachment From <email address hidden> 2015-06-11 12:21 EDT-------

Can you add this patch into grub package? I'm planning to push it upstream soon.

Adam Conrad (adconrad)
affects: debian-installer (Ubuntu) → grub2 (Ubuntu)
Adam Conrad (adconrad)
Changed in grub2 (Ubuntu):
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.02~beta2-25

---------------
grub2 (2.02~beta2-25) unstable; urgency=medium

  * Build-depend on dosfstools and mtools on arm64 as well as amd64.

 -- Colin Watson <email address hidden> Fri, 12 Jun 2015 12:29:19 +0100

Changed in grub2 (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Current wily build exception on powerkvm

------- Comment (attachment only) From <email address hidden> 2015-06-15 06:42 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-06-15 12:49 EDT-------
I updated my machine with grub2 package 2.02~beta2-25 here and it works fine. I build the same version and it works fine too. I'm not sure if this is really a problem in grub2 binary.

Besides this exception, we have an installation failure since Jun 11:

Jun 15 10:06:09 in-target: The following information may help to resolve the situation:
Jun 15 10:06:09 in-target:
Jun 15 10:06:09 in-target: The following packages have unmet dependencies:
Jun 15 10:06:09 in-target: libpam-systemd : Depends: systemd (= 219-10ubuntu1) but 220-6ubuntu1 is to be installed
Jun 15 10:06:09 in-target: E
Jun 15 10:06:09 in-target: : Unable to correct problems, you have held broken packages.
Jun 15 10:06:09 in-target: tasksel: aptitude failed (100)
Jun 15 10:06:09 main-menu[232]: WARNING **: Configuring 'pkgsel' failed with error code 1
Jun 15 10:06:09 main-menu[232]: WARNING **: Menu item 'pkgsel' failed.

------- Comment From <email address hidden> 2015-06-15 12:52 EDT-------
Just FYI that I pushed the fix upstream today:
http://git.savannah.gnu.org/cgit/grub.git/commit/?id=afd0f21b2027310fda52b00ac1b964041d39a363

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Well, that particular failure is unrelated to grub -- it's caused by systemd updates which fail the installation while all packages aren't up to date on the mirrors being used.

I've noticed the fix is on the upstream git as well as in Debian; we'll get it in Ubuntu shortly.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-06-18 13:28 EDT-------
Canonical, can you please check if the GRUB used by the daily image is updated? I opened the ISO file from today and the GRUB elf is from May 31:

total 274K
-r--r--r-- 1 root root 294 May 31 16:02 grub.cfg
-r--r--r-- 1 root root 272K May 31 16:02 powerpc.elf
dr-xr-xr-x 2 root root 2.0K May 31 16:02 powerpc-ieee1275

Moreover, the installation is failing, as I said in the previous comment, on the libpam-systemd package.

------- Comment From <email address hidden> 2015-06-22 12:28 EDT-------
Good news!

Both problems were fixed and we have a working wily again. Please test it.

------- Comment From <email address hidden> 2015-06-22 12:37 EDT-------
Working for PowerVM as well....

PowerPC Firmware
Version SV810_101
SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.
-------------------------------------------------------------------------------
Current Boot Sequence
1. SCSI CD-ROM
( loc=U8247.21L.21239AA-V3-C2-T1-L8200000000000000 )
2. None
3. None
4. None

GNU GRUB version 2.02~beta2-26

+----------------------------------------------------------------------------+
|*Install |
| Rescue mode |

------- Comment From <email address hidden> 2015-06-22 20:45 EDT-------
I was able to successfully install and reboot 15.10 from both virtual CD-ROM (daily ISO downloaded today) and from the netboot image downloaded today on two different PowerKVM 3.1 guests.

This bug looks fixed to me.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers