[xen] The kernel panics on transfer

[Krzysztof Leszczyński]

I got a 1GB network card and I tried to copy my xen image (30GB), so I run
  nc -l 3000 > image.dd
on xen dom0
and tried to send using nc from another remote machine.

The machine crashed after 3-15 seconds. I tried this many many times.
I managed to get bug using netconsole (attached).

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-32-generic 3.13.0-32.57
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Aug 21 15:06 seq
 crw-rw---- 1 root audio 116, 33 Aug 21 15:06 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg:
 [ 13.914872] init: plymouth-upstart-bridge main process ended, respawning
 [ 98.312437] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
Date: Thu Aug 21 15:09:13 2014
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:en
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: placeholder root=/dev/mapper/r5-system1 ro biosdevname=0 nosplash netconsole=4444@192.168.10.200/eth1,3000@192.168.10.29/bc:ae:c5:e0:af:6c
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-32-generic N/A
 linux-backports-modules-3.13.0-32-generic N/A
 linux-firmware 1.127.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/26/2006
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 080010
dmi.board.name: TYAN High-End Dual AMD Opteron, S2882
dmi.board.vendor: TYAN
dmi.board.version: To be filled by O.E.M.
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr080010:bd07/26/2006:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnTYAN:rnTYANHigh-EndDualAMDOpteron,S2882:rvrTobefilledbyO.E.M.:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue occur in a previous version of Ubuntu, or is this a new issue?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.17 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc1-utopic/

[Krzysztof Leszczyński (chris-lesz)]

Re: Joseph, this machine worked perfectly with ubuntu 12.04/xen and never crashed in last 1.5 year (just maintenance reboots for upgrades).

Yes, it crashes on 3.17-rc1 on xen but it works without problems when run without xen on bare metal.
On xen it is not able to load even a single module and leaves itself in initrd shell. The very same kernel works perfectly (with the same initrd file) when booted directly.

I'm attaching the bootlog.

[Stefan Bader (smb)]

Jo, Kzysztof, 3.16/3.17 currently suffer from a change affecting the kernel/module split. I am trying to get behind this. But right now now PV guest works without re-compiling the kernel with CONFIG_RANDOMIZE_BASE_MAX_OFFSET=0x20000000 (resulting in the same layout as before).

The stack trace itself does not directly contain Xen specific functions (actually that is not a surprise since this is transferring to dom0, which uses the real NIC). So we are talking about an issue related to receiving large transfers on a NetXtreme BCM5704 (that would be the tg3 driver). Comment #4 is a bit confusing. I assume it meant a 3.17 kernel works on bare metal but not as dom0 which would be the max offset problem. The other question would be: does the file transfer work, when not booted as dom0?

Maybe as a first step for further debugging I would try to upgrade the dom0 to the most current Trusty kernel (3.13.0-34.60) or maybe the proposed version manually from [1]. That would be 3.13.0-35.62 (you would need linux-headers (arch and all) linux-image (arch), and linux-image-extra (arch). For checking 3.17, we need a special kernel.

[1] http://archive.ubuntu.com/ubuntu/pool/main/l/linux/

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]