reboot under Oracle Virtual Box not possible

[Christian Bachmaier]

Running 16.04 server without X under latest Oracle Virtual Box 5.0.12 on Windows. After issuing 'shutdown -r now' the following boot process is not successful and hangs with message:
---[ end of Kernel panic - not syncing: VFS: unable to mound root fs on unknown-block(0,0)

If using 'shutdown -h now' and then staring the machine anew from Virtual Box manager, then the system starts as it should. Thus, for short, warm reboot hangs and cold reboot operates as desired. I have the feeling that this behavior was introduced with kernel 4.3, but I cannot swear on it.

WORKAROUND: Switch network adapter type to Intel Pro/1000 MT Desktop from paravirtualized network (virtio-net).

---
AlsaDevices:
 total 0
 crw-rw----+ 1 root audio 116, 1 Jan 6 11:34 seq
 crw-rw----+ 1 root audio 116, 33 Jan 6 11:34 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.19.3-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=UUID=cbc2edd0-9886-40c6-bf14-d75f638fc4cf
InstallationDate: Installed on 2015-12-23 (14 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Alpha amd64 (20151115)
IwConfig:
 enp0s3 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
Package: linux (not installed)
PciMultimedia:

ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.3.0-5-generic root=UUID=07e383b5-7e12-4530-8ae0-75c104f67e2b ro
ProcVersionSignature: Ubuntu 4.3.0-5.16-generic 4.3.3
RelatedPackageVersions:
 linux-restricted-modules-4.3.0-5-generic N/A
 linux-backports-modules-4.3.0-5-generic N/A
 linux-firmware 1.154
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
Uname: Linux 4.3.0-5-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1531455

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[Christian Bachmaier (chkis)]

apport information

[penalvch (penalvch)]

Christian Bachmaier, thank you for reporting this and helping make Ubuntu better.

At your earliest convenience, could you please test the latest upstream kernel available from the very top line at the top of the page from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D (the release names are irrelevant for testing, and please do not test the daily folder)? Install instructions are available at https://wiki.ubuntu.com/Kernel/MainlineBuilds . This will allow additional upstream developers to examine the issue.

If testing on your main install would be inconvenient, one may:
1) Install Ubuntu to a different partition and then test this there.
2) Backup, or clone the primary install.

If the latest kernel did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this issue is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where X, and Y are the first two numbers of the kernel version, and Z is the release candidate number if it exists.

If the mainline kernel does not fix the issue, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Please note, an error to install the kernel does not fit the criteria of kernel-bug-exists-upstream.

Once testing of the latest upstream kernel is complete, please mark this report's Status as Confirmed. Please let us know your results.

Thank you for your understanding.

[Christian Bachmaier (chkis)]

With latest Kernel 4.4-rc8 exactly the same problem. Warm/Reboot hangs and cold boot works.

Here the latest output from the serial device I could grab after starting the kernel with console=ttyS0,115200n8 . Maybe the first supect entry is [ 37.977527] RAMDISK: incomplete write (28969 != 32768) .

[ 32.622294] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 32.739918] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 *11)
[ 32.814085] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 *11)
[ 32.886389] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 *10 11)
[ 32.972507] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *9 10 11)
[ 33.058506] ACPI: Enabled 2 GPEs in block 00 to 07
[ 33.130136] vgaarb: setting as boot device: PCI:0000:00:02.0
[ 33.222488] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 33.360394] vgaarb: loaded
[ 33.402376] vgaarb: bridge control possible 0000:00:02.0
[ 33.488681] SCSI subsystem initialized
[ 33.517301] ACPI: bus type USB registered
[ 33.578541] usbcore: registered new interface driver usbfs
[ 33.663454] usbcore: registered new interface driver hub
[ 33.745715] usbcore: registered new device driver usb
[ 33.833963] PCI: Using ACPI for IRQ routing
[ 33.877871] NetLabel: Initializing
[ 33.934345] NetLabel: domain hash size = 128
[ 34.001805] NetLabel: protocols = UNLABELED CIPSOv4
[ 34.075036] NetLabel: unlabeled traffic allowed by default
[ 34.163253] clocksource: Switched to clocksource kvm-clock
[ 34870]ppmo AArr lest Eable
  5.71] p:nPCP fnd dic
[ 35.548313] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 35.552576] NET: Registered protocol family 2
[ 35.553011] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[ 35.553576] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[ 35.556144] TCP: Hash tables configured (established 16384 bind 16384)
[ 35.556657] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[ 35.559067] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[ 35.562046] NET: Registered protocol family 1
[ 35.562389] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 35.565144] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 35.567385] Trying to unpack rootfs image as initramfs...
[ 35.792364] rootfs image is not initramfs (junk in compressed archive); looks like an initrd
[ 35.877874] Freeing initrd memory: 29664K (ffff880034600000 - ffff8800362f8000)
[ 35.948245] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 36.008763] RAPL PMU detected, API unit is 2^-32 Joules, 4 fixed counters 10737418240 ms ovfl timer
[ 36.082997] hw unit of domain pp0-core 2^-0 Joules
[ 36.114337] hw unit of domain package 2^-0 Joules
[ 36.147513] hw unit of domain dram 2^-0 Joules
[ 36.169050] hw unit of domain pp1-gpu 2^-0 Joules
[ 36.192590] Scanning for low memory corruption every 60 seconds
[ 36.233018] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 36.295850] audit: initializing netlink subsys (disabled)
[ 36.314798] audit: type=2000 audit(1452151393.649:1): initialized
[ 36.315458] Initialise syst...

[penalvch (penalvch)]

Christian Bachmaier, to rule out a regression, could you please test for this with an earlier mainline version (ex. 3.13.x) and advise to the results?

[Christian Bachmaier (chkis)]

So I tried kernels 4.2.8, 4.1.0 and 3.13.11.11 . All show the same beaviour: kernel panic after warm reboot. However now I played with the setting of Virtual Box and switched back the network adapter type paravirtualized network (virtio-net) from to Virtual Box's default Intel Pro/1000 MT Desktop OEM). As far I can remember this was probably before the update to kernel 4.3 from 4.2. Thus the latter has to say nothing. As the network performance seems to be roughly the same, this solves the bug for me, or is at least a reasonable workaround for a productive machine.

Thus all looks like it is really a bug in Virtual Box as I have indicated in my first posting?

Thanx for your great help! Should I or can I close this bug then, or switch it to the Virtual Box Team, or Christopher, could you do this?

Chris

[Christian Bachmaier (chkis)]

Now I am confused, I already have tested 3.13.x with 3.13.11.11 and on shutdown -r now it hangs with kernel panic when using virtio-net of Oracle Virtual Box instead the HW Emulation of the Intel card. See my previous post. Same with the newest kernel.

[Christian Bachmaier (chkis)]

When using Vurtual Box's Intel Pro/1000 MT Desktop emulation then everything works with all kernels I have tested so far.

[penalvch (penalvch)]

Switching over to virtualbox, as it appears an issue with their kernel module (i.e. not linux kernel issue).

[Gianfranco Costamagna (costamagnagianfranco)]

Hi, upstream is aware and working on a fix.
thanks for reporting it!

[penalvch (penalvch)]

LocutusOfBorg, thanks for the quick follow up. Is there an upstream public report, or mailing list discussion that could be referenced to this?

[Gianfranco Costamagna (costamagnagianfranco)]

They told me they tracked it in their internal issue tracker.
They didn't give me any ETA, so I asked to be notified on fix done (probably in the next release I hope)

[Alex (alexey-strelnikov)]

Hello!

I was unable to reproduce 'shutdown -r' hangs with kernel panic with similar setup like described here (VBox 5.0.12, virtio-net, Ubuntu 16.04 latest nightly build). Would be good to get more information about host/guest config or even better way is attach/share somewhere guest appliance in OVA format (File->Export Appliance)
Thanks!

[Christian Bachmaier (chkis)]

Hi,

as I wrote in my first comment, my host is a M$ Windows system, on which I prepare the vms to run later on a dedicated Solaris virtualization server. My first thought was it is a kernel problem. However, kernel panic still is reproducible with todays version of 16.04 as guest (all updates included) and VirtualBox 5.0.14 r105127 when using virtio-net and warm boot. My network config is in bridge mode to the physical adapter of the Windows host. Chipset emulation PIIX3, IO-APIC active, no audio, VT-x/AMD-V on, Nested Paging, KVM-paravitualization on, template Ubuntu (64-bit), SATA controller in AHCI mode, hard disk as vmdk file.

I assume your host is Linux... maybe that's the key. Then Ubuntu seems to have no problem, neither as guest nor as host. At least good to know.

Sorry that I cannot provide an ova: This is a copy of my production system with the new LTS version for testing purposes containing a lot of sensible data. But basically the guest is a 16.04 standard server installation without X.

Hope that helps at least a bit.

LocutusOfBorg wrote:
> They told me they tracked it in their internal issue tracker.
Hmm, ...

Chris

[Alex (alexey-strelnikov)]

Hi Chris,

I tested this issue on Windows 7, 10 and Ubuntu 15.10 hosts, NAT and bridge networking using virtio-net adapter.
I think guest configuration is also important. Do you have any CIFS/NFS/etc shares mounted in the guest? Sounds like 'Unable to mount/umount root fs' and networking could be the key.

Regards, Alex.

[Christian Bachmaier (chkis)]

Hi Alex,

my backup folder to a zfs filesystem is mounted via NFS automount. Here is my config:

$ cd /etc/auto.master.d/
$ cat backup.auto
hanuta -fstype=nfs,rw,sloppy backupserver.mycompany.de:/vmdata/extern/hanuta
$ cat backup.autofs
/backup /etc/auto.master.d/backup.auto
$

However, I have no script or similiar which changes into the directory /backup/hanua during boot. So it will not be mounted and thus is probably not causal. Another thing is that backupserver is not reachable from my windows box, but only from the production server due to security restrictions.

Thanks,

Chris

[Ivan Vilata i Balaguer (ivilata)]

Hi, I found a similar problem with a Debian Jessie VM with kernel 3.16.0-4-686-pae, so the situation may not be completely applicable to Ubuntu 16.04, but I think it may help.

I found that unloading just the virtio_net module before rebooting made the kernel panic not happen (no need to change NIC driver to Intel).

So as a temporary workaround, if your VM network interface is eth0, you may make its entry in /etc/newtwork/interfaces look like this:

    iface eth0 inet dhcp
        pre-up modprobe virtio_net
        post-down modprobe -r virtio_net

This worked for me and can be done completely inside of the VM.

I hope that this issue gets fixed since it may be a showstopper for remotely managed VMs. Thanks!

[Gianfranco Costamagna (costamagnagianfranco)]

Hello does this still happen with 5.1.22?

[Ivan Vilata i Balaguer (ivilata)]

Yes, I noticed this with 5.1.18_Debian r114002 and I checked that it still happens with 5.1.22_Debian r115126, and my workaround still works as well.

[Gianfranco Costamagna (costamagnagianfranco)]

Can you please report on vbox-dev mail list? mentioning their internal issue tracker?

thanks

[Frank Mehnert (frank-mehnert-q)]

If we only could reproduce this problem. Tried Debian 8 and Ubuntu 16.04, both with virtio-net. With and without (only multi-user) graphical desktop: Guest reboot works always, I never saw any guest panic.