Can't boot Citrix netscaler image

[Nikita Gubenko]

There is a problem with booting Netscaler image (freebsd based) with MOS 6.0/5.1 (ubuntu). Seems like it dies on attaching keyboard:

atkbd0: <AT Keyboard> irq 1 on atkbdc0

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff808d59ee
stack pointer = 0x28:0xffffffff96376800
frame pointer = 0x28:0xffffffff96376850
code segment = base 0x0, limit 0xfffff, type 0x1b
   = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, IOPL = 0
current process = 0 (swapper)
cpu 0 [thread pid 0 tid 100000 ]
Stopped at get_typematic+0x7e: movzwl (%rax),%eax

I was able to workaround this by replacing seabios on the computes with newer version (replaced /usr/share/seabios/bios-256k.bin with https://github.com/qemu/qemu/raw/master/pc-bios/bios-256k.bin)

Image is here https://citrix.sharefile.com/d/s397e9fccace48009

[Pavel Boldin (pboldin)]

Please provide sha256 or md5 hashes along the files.

[Nikita Gubenko (nikita-gubenko)]

md5 8cbba106ae5036a999e2b50fb2e8d5d2

[OSCI Robot (oscirobot)]

RPM package seabios has been built for project packages/centos6/seabios
Package version == 1.8.1, package release == 1

Changeset: https://review.fuel-infra.org/5293
project: packages/centos6/seabios
branch: 5.1
author: Pavel Boldin
committer: Pavel Boldin
subject: Updating to 1.8.1 fixing FreeBSD boot
status: patchset-created

Files placed on repository:
seabios-1.8.1-1.mira1.noarch.rpm
seabios-bin-1.8.1-1.mira1.noarch.rpm
seavgabios-bin-1.8.1-1.mira1.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-5293/centos

[OSCI Robot (oscirobot)]

DEB package seabios has been built for project packages/precise/seabios
Package version == 1.8.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/5144
project: packages/precise/seabios
branch: 5.1
author: Pavel Boldin
committer: Pavel Boldin
subject: Updating to 1.8.1 fixing FreeBSD boot
status: patchset-created

Files placed on repository:
seabios_1.8.1-ubuntu1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-5144/ubuntu

[Pavel Boldin (pboldin)]

The above did not fixed the problem under the Ubuntu, so I'm looking for the reasons for this.

[Pavel Boldin (pboldin)]

This is seems to be caused by the Ubuntu toolchain that generates an incorrect binaries.

If I copy `seabios-1.8.1' (or even 1.7.4) built by CentOS to the Ubuntu host it works fine. With `seabios-1.8.1' build by Ubuntu toolchain FreeBSD fails to run.

[OSCI Robot (oscirobot)]

DEB package seabios has been built for project packages/precise/seabios
Package version == 1.8.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/5144
project: packages/precise/seabios
branch: 5.1
author: Pavel Boldin
committer: Pavel Boldin
subject: seabios-1.8.1 built by gcc-4.5 fixing FreeBSD boot
status: patchset-created

Files placed on repository:
seabios_1.8.1-ubuntu1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-5144/ubuntu

[Pavel Boldin (pboldin)]

6.1 is not affected since it uses Ubuntu Trusty as a base distro.

[OSCI Robot (oscirobot)]

DEB package seabios has been built for project packages/precise/seabios
Package version == 1.8.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/5557
project: packages/precise/seabios
branch: 6.0-updates
author: Pavel Boldin
committer: Pavel Boldin
subject: seabios-1.8.1 built by gcc-4.5 fixing FreeBSD boot
status: patchset-created

Files placed on repository:
seabios_1.8.1-ubuntu1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5557/ubuntu

[Pavel Boldin (pboldin)]

I've built a custom ISO with the fixes presented above for the 6.0

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6.0_iso/51/

And it have passed BVTs:

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/old.custom.ubuntu.bvt_2/
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/old.custom.centos.bvt_1/

[Pavel Boldin (pboldin)]

I've built a custom ISO with the fixes presented above for the 5.1

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_5.1_iso/58/

And it have passed BVTs:

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/old.custom.centos.bvt_1/1/
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/old.custom.ubuntu.bvt_2/1/

[Pavel Boldin (pboldin)]

Committed fixes are to be cherry-picked to a side branches after merging the mainline.

[Pavel Boldin (pboldin)]

RCA is the following.

Analysis:
QEMU running `seabios' from the package of versions 1.7.4 and 1.8.1 fails to boot FreeBSD if the `seabios' binaries were compiled with gcc-4.6.3 provided by Ubuntu Precise.

The `seabios' package compiled by default CentOS 6 compiler works fine. The `seabios' package compiled by default Ubuntu Trusty compiler works fine thus rendering Fuel 6.1 unaffected.

Solution:
The `seabios' package for Ubuntu Precise was updated to version 1.8.1 and is now compiled by the `gcc-4.5'. This fixes the FreeBSD boot issue and is not introducing any regressions.

Follow-up:
In case the real root cause is required we should further analyse `seabios' packages built by different gcc versions under Ubuntu Precise.

[Pavel Boldin (pboldin)]

I would like someone to exactly point releases we want this fix in. Should it be only 6.0-updates and 5.1.1-updates?

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5557
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.0-updates

Commit: 9d8aa61d522cccdf5d7d5423cbfe1d8a0692f343
Author: Pavel Boldin <email address hidden>
Date: Thu Apr 9 14:03:29 2015

seabios-1.8.1 built by gcc-4.5 fixing FreeBSD boot

Change-Id: Ib7a011aee36e9b852cfe0e467985d60a8924ac54
Closes-Bug: #1435501

[Vitaly Sedelnik (vsedelnik)]

Won't Fix in 5.1-updates and 5.1.1-updates because of Medium priority.