unable to ssh to guest vm after volume attach when using virtio devices

[Jason Rogers]

Binary package hint: qemu-kvm

Description: Ubuntu 10.04 LTS
Release: 10.04

kvm:
  Installed: 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9
  Candidate: 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9
  Version table:
 *** 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

I had a guest vm running on a eucalyptus cloud using virtio for disk and network I/O. I also had an ssh session on the guest vm. When I attach a volume to the instance using euca-attach-volume I lose network connectivity to the guest.

Before I upgraded to lucid I was running karmic, and there was no issue. After the upgrade, I started having issues.

I've tried the UEC 9.04, 9.10 and 10.04 i386 images, along with some custom amd64 images that worked previously under karmic.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: qemu-kvm 0.12.3+noroms-0ubuntu9
ProcVersionSignature: Ubuntu 2.6.32-21.32-server 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-server x86_64
Architecture: amd64
Date: Fri May 7 10:27:15 2010
ExecutablePath: /usr/bin/qemu-system-x86_64
KvmCmdLine:
 UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
 root 3992 1 2 180723 137728 0 10:12 ? 00:00:22 /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 1 -name i-3B7A06D0 -uuid eddba550-57a7-ecf7-5201-6cfea02ab82c -nographic -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-3B7A06D0.monitor,server,nowait -monitor chardev:monitor -boot c -kernel /var/lib/eucalyptus/instances/admin/i-3B7A06D0/kernel -initrd /var/lib/eucalyptus/instances/admin/i-3B7A06D0/ramdisk -append root=/dev/vda1 console=ttyS0 -drive file=/var/lib/eucalyptus/instances/admin/i-3B7A06D0/disk,if=virtio,index=0,boot=on -net nic,macaddr=d0:0d:3b:7a:06:d0,vlan=0,model=virtio,name=virtio.0 -net tap,fd=47,vlan=0,name=tap.0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances/admin/i-3B7A06D0/console.log -serial chardev:serial0 -parallel none -usb
Lsusb:
 Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Foxconn OEM
ProcAttrCurrent: libvirt-eddba550-57a7-ecf7-5201-6cfea02ab82c (enforce)
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.32-21-server root=/dev/mapper/hostname-root ro acpi=force pci=nommconf quiet splash
ProcEnviron: PATH=(custom, no user)
SourcePackage: qemu-kvm
dmi.bios.date: 02/10/2007
dmi.bios.vendor: Phoenix Technologies, LTD
dmi.bios.version: 6.00 PG
dmi.board.name: K8M890-8237A
dmi.board.vendor: Foxconn
dmi.chassis.type: 3
dmi.chassis.vendor: OEM
dmi.modalias: dmi:bvnPhoenixTechnologies,LTD:bvr6.00PG:bd02/10/2007:svnFoxconn:pnOEM:pvr:rvnFoxconn:rnK8M890-8237A:rvr:cvnOEM:ct3:cvr:
dmi.product.name: OEM
dmi.sys.vendor: Foxconn

[Jason Rogers (juk80x)]

Additionally, if I start the guest vm without virtio enabled in kvm, then there are no issues.

[Torsten Spindler (tspindler)]

On Fri, 2010-05-07 at 14:35 +0000, Jason Rogers wrote:
...
> I had a guest vm running on a eucalyptus cloud using virtio for disk and
> network I/O. I also had an ssh session on the guest vm. When I attach a
> volume to the instance using euca-attach-volume I lose network
> connectivity to the guest.

With what arguments did you call euca-attach-volume? I believe there is
a problem when specifying /dev/sda instead of /dev/sdb.

[Jason Rogers (juk80x)]

I was using virtio so the argument was "-d /dev/vdb"

[Scott Moser (smoser)]

using virtio is definitely not a supported/tested path. As far as I know, neither are even configurable other than by hacking files on the node.

If this is reproducible, could you please attach the console output of the instance (euca-get-console-output).

[Jason Rogers (juk80x)]

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.32-21-generic-pae (buildd@rothera) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #32-Ubuntu SMP Fri Apr 16 09:39:35 UTC 2010 (Ubuntu 2.6.32-21.32-generic-pae 2.6.32.11+drm33.2)
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] NSC Geode by NSC
[ 0.000000] Cyrix CyrixInstead
[ 0.000000] Centaur CentaurHauls
[ 0.000000] Transmeta GenuineTMx86
[ 0.000000] Transmeta TransmetaCPU
[ 0.000000] UMC UMC UMC UMC
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000001fffb000 (usable)
[ 0.000000] BIOS-e820: 000000001fffb000 - 0000000020000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)
[ 0.000000] DMI 2.4 present.
[ 0.000000] last_pfn = 0x1fffb max_arch_pfn = 0x1000000
[ 0.000000] x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] modified physical RAM map:
[ 0.000000] modified: 0000000000000000 - 0000000000002000 (usable)
[ 0.000000] modified: 0000000000002000 - 0000000000006000 (reserved)
[ 0.000000] modified: 0000000000006000 - 000000000009f000 (usable)
[ 0.000000] modified: 000000000009f000 - 00000000000a0000 (reserved)
[ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] modified: 0000000000100000 - 000000001fffb000 (usable)
[ 0.000000] modified: 000000001fffb000 - 0000000020000000 (reserved)
[ 0.000000] modified: 00000000fffbc000 - 0000000100000000 (reserved)
[ 0.000000] init_memory_mapping: 0000000000000000-000000001fffb000
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] ACPI: RSDP 000f8940 00014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 1fffde30 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 1ffffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 1fffdfd0 01E22 (v01 BXPC BXDSDT 00000001 INTL 20090123)
[ 0.000000] ACPI: FACS 1ffffe00 00040
[ 0.000000] ACPI: SSDT 1fffdf90 00037 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: APIC 1fffdeb0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] ACPI: HPET 1fffde70 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
[ 0.000000] 0MB HIGHMEM available.
[ 0.000000] 511MB LOWMEM available.
[ 0.000000] mapped low ram: 0 - 1fffb000
[ 0.000000] low ram: 0 - 1fffb000
[ 0.000000] node 0 low ram: 00000000 - 1fffb000
[ 0.000000] node 0 bootmap 00009000 - 0000d000
[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 001fffb000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
[ 0.000000] #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [000...

[Mathias Gug (mathiaz)]

How did you configure the NC to use virtio instead of the default block device?

Is the guest still running after attaching the device?

[Jason Rogers (juk80x)]

I modified /usr/share/eucalyptus/gen_kvm_libvirt_xml as follows,

95c95
< <cmdline>root=/dev/sda1 console=ttyS0</cmdline>
---
> <cmdline>root=/dev/vda1 console=ttyS0</cmdline>
105c105
< <target dev='sda'/>
---
> <target dev='vda' bus='virtio'/>
110c110
< <model type='e1000'/>
---
> <model type='virtio'/>

The guest is still running. I was even able to get console output. But there is no ssh access.

[Launchpad Janitor (janitor)]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]