Nexenta fails to run as Xen domU

[Justin Thomas]

I'm attempting to install NCP 2.0 RC1 (b104) as a paravirtualized guest on XenServer 5.0. I'm using a method similar to what I have used to successfully boot OpenSolaris (from 2008.11 to the most recently released snv_109).

The kernel seems to boot okay (both 32-bit and 64-bit kernels), but fails to find the ZFS filesystem:

v3.2.1 chgset '58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)'
SunOS Release 5.11 Version NexentaOS_20081207 64-bit
Loading Nexenta...
NOTICE: xdf@51712: failed to read feature-barrier
NOTICE: zfs_domount: error 19
Cannot mount root on syspool/rootfs-nmu-000 fstype zfs

panic[cpu0]/thread=fffffffffbc60020: vfs_mountroot: cannot mount root

fffffffffbc92de0 genunix:vfs_mountroot+350 ()
fffffffffbc92e10 genunix:main+f0 ()
fffffffffbc92e20 unix:_locore_start+80 ()

skipping system dump - no dump device configured
rebooting...

The XenServer is configured as follows:

PV-kernel is pointed at the /platform/i86xpv/kernel/amd64/unix 64-bit kernel copied to the XenServer as /opt/kernels/kernel_nexenta_104_x64. Similarly, PV-ramdisk is pointed at the /platform/i86pc/amd64/boot_archive copied to the XenServer as /opt/kernels/ramdisk_nexenta_104_x64.

xe vm-param-set uuid=[VM UUID] PV-kernel=/opt/kernels/kernel_nexenta_104_x64
xe vm-param-set uuid=[VM UUID] PV-ramdisk=/opt/kernels/ramdisk_nexenta_104_x64

PV-args is set as follows:

xe vm-param-set uuid=[VM UUID] PV-args='/platform/i86xpv/kernel/amd64/unix -B console=ttya,zfs-bootfs=syspool/rootfs-nmu-000,bootpath="/xpvd/xdf@51712:a"'

HVM-boot-policy is set to null to force the system into PV mode for this VM.

This method is almost identical to that successfully used for OpenSolaris as described in my blog at:

http://justindthomas.wordpress.com/2008/12/22/xenserver-5-and-opensolaris-0811/

Note that I would normally try to use the PV kernel to boot from the CD to install before actually running through an entire HVM install as I did in this case (installing and operating via HVM is horrendously slow). Unfortunately, there is no i86xpv kernel on the CD to refer to in the boot parameters; perhaps that is key to this issue as well. When I attempt that (using the "install_media=cdrom" and removing the zfs-boots and bootpath boot parameters), I get this:

.2.1 chgset '58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)'
SunOS Release 5.11 Version NexentaOS_20081207 64-bit
Loading Nexenta...
NOTICE: mount: not a UFS magic number (0x0)
Cannot mount root on /ramdisk:a fstype ufs

panic[cpu0]/thread=fffffffffbc60020: vfs_mountroot: cannot mount root

fffffffffbc92de0 genunix:vfs_mountroot+350 ()
fffffffffbc92e10 genunix:main+f0 ()
fffffffffbc92e20 unix:_locore_start+80 ()

[Erast (erast)]

This will not be fixed in 2.0. Too late. But we will target it for 2.1 (this summer)

[Justin Thomas (justin-jdthomas)]

I did discover some workarounds. In this case, modifying the miniroot archive to include the PV kernel and changing the boot arguments to force the system into maintenance mode allowed me to force import the zpool from the HVM install. Once that was done, I could change the boot arguments back and the system was able to successfully boot. See http://justindthomas.wordpress.com for details.

Thanks!