apparmor: cannot successfully start an Eucalyptus KVM instance

[C de-Avillez]

Started testing Euca 2.0. Ran a first batch of 200 instances, *all* failed (still to analyse the log). I then cleaned up the install, and bounced eucalyptus CLEAN=1 just in case; then I added a keypair, a group, and set IP tables for it; then I started one instance. The instance stayed in PENDING for a while, then moved to TERMINATED. Logs attached.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: eucalyptus-cloud 2.0~bzr1211-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.35-10.15-server 2.6.35-rc5
Uname: Linux 2.6.35-10-server x86_64
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="UEC-TEST1"
Architecture: amd64
Date: Mon Jul 26 18:32:30 2010
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus

[C de-Avillez (hggdh2)]

command sequence:

ubuntu@cempedak:~/uec-testing-scripts/results$ . ~/eucarc
ubuntu@cempedak:~/uec-testing-scripts/results$ euca-add-keypair --config=users/admin//eucarc uectest-k0
KEYPAIR uectest-k0 c1:37:a0:13:55:d5:6c:18:9e:88:84:6f:99:3f:fa:ef:df:52:be:d6
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAvEeXpF0FnzOOFy1V/FgdwFE+MgOFMbOIwBzhUd/NSiQWZmVp
r5RxBIBERdXzoJ+CLgwnv4S1kC9PTiiGzTTncji0DgVq19a75fOiAPIL2KLEAqds
uCXndmMx2KVw7PDhB0ntlFSqZbFs4sTeiWfaojmiVp/UoSeh69lrjWpTFJJTI/5A
adq9XAjK/93FGeveqIbcJIa3bsaYXIv+vAkMlqIF0/zabO3xmHLk9SfC5ylIm6q2
w15EvbhU+KX0RuQtn9xh64cf6MlkmdO7TQzrLF83tKYr02nX0LzWAF7G1FBTmjBs
tiTxTi9eK26xXBGLhb/pema9X4Bd7pDiei3jqwIDAQABAoIBAGXJ8sKqg8qRDcWb
/fuiC9uYUJ6rZov7OvhZ5Lrl2s+1eOK8+wkwoHVFfyOeSstQ0QliCu8wqRQdb6ua
n147ttD6tOVLqnqsHgNw1unZ8euZ+4xPxag83ckUUbOFpeqDQ7U0E5x4tXKrHGnw
8zqoCfxgWlKosES7739ZmRroG6hnjxvjy3tyXTCh+og7qGarNw5FIe+rUrwjGc/K
8PykBmPcAPYdMDQPOI3L6W7CWHmEIssVFQIvJIhDj2069XyWM7N9nZ30cr/b4jaV
8GpEsrX2it+4v8dQzV6ss519kdLNcPYFsV8MTtI/6vzv44XEoqlZbDa+IGixiTE4
zvZq0hkCgYEA/+RpJSZ6HJRpHJ9eMWIMjzENa6LJq+CN7A79Fc55zscyP5ZqPnax
CKU6LTXhmf/OVTykfr5Xxusr0mLACMTiZChIzdQdRfUC40D3lIZvRLZEYQ19WWqR
Nz5D3PwpV1o5yVG4Yn7ow+2vV7VZjg9MF+yDbrLzEVL/tzp0eNrx7Z0CgYEAvFvk
VFAATYXFY4zTAqVmY7zHN+1Df38t8Z+U7h5nDdXHEI1PZaeWXLMOs2+fs40dLAmo
e/USHntGDu52lHfBquCZ75xAF1nI9hrQq6IkQggvMYCK+ztqne51HMBec4cR7MZx
2E/6EuY+E/7NARkFwPPvWkG83t97ZjKec9C29+cCgYBwgW4xJT+e/SlRqkfxklhP
RJPP9RyR8wKy/MwNIGCxpybSikFgATNIx8xSQFRWLv/6vdTc3LcfABqeP0iLEnUu
7/9Cb7YHIfgJDnFL3edh4Hkj6phoDdCW50tTB5Ltkv/BIards0qHiiMDfvpG2Tbt
ddE7hHYV8dX4Kz02PZ0MiQKBgHF11OsoqZYI72bQ7EZaxXtsuoAozpDdVhqG6xoo
fCEsVdXVmS79rSg3v7I8mfF5HkppovDQJdLwpLnX0NwL6z/4PSHFeCHYtqHwcyP/
7mFMNslVyeId31I00py4D+z9PsHmo6sOOe/qexhmUqzHmtyo09QFZzTFX7qm4SWY
Yq6fAoGATV3c3srphCuSOxKSuznJ0Ri8m8RIc1DE1tGEqm/kj6/evOYEbSJ5Xa6h
uIau4vmgcaVJJ57MVKu8ZwzjD72MRf1VhcUDPY9ErfHQAIJKFkzf/5xZw9CmbxoS
KNotj+uatEnKmRCIoLgy1vF2q73dLOVTYBePhIOsDuZ/RkbZKVE=
-----END RSA PRIVATE KEY-----

ubuntu@cempedak:~/uec-testing-scripts/results$ cd ../users/admin
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ ls
admin.zip cloud-cert.pem euca2-admin-528d5aa8-cert.pem euca2-admin-528d5aa8-pk.pem eucarc jssecacerts uectest-k0.priv
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ vi uectest-k0.priv
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ euca-add-group --config=users/admin//eucarc -d UEC-test uectest-g0
GROUP uectest-g0 UEC-test
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ euca-describe-groups
GROUP admin default default group
GROUP admin uectest-g0 UEC-test
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ euca-authorize --config=users/admin//eucarc uectest-g0 -P tcp -p 22 -s 0.0.0.0/0
GROUP uectest-g0
PERMISSION uectest-g0 ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0
ubuntu@cempedak:~/uec-testing-scripts/users/admin$ euca-run-instances --config=users/admin//eucarc emi-E86B1B3F -k uectest-k0 -g uectest-g0 -t m1.xlarge
RESERVATION r-34DA06A5 admin admin-uectest-g0
INSTANCE i-417B0807 emi-E86B1B3F 0.0.0.0 0.0.0.0 pending uectest-k0 0 m1.xlarge 2010-07-26T22:27:31.712Z UEC-TEST1 eki-77A3204C
ubuntu@cempe...

[C de-Avillez (hggdh2)]

No, this looks like an apparmour issue:

ubuntu@mabolo:/var/log/libvirt/qemu$ sudo cat i-417B0807.log
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name i-417B0807 -uuid 2c2cd7ed-0a52-48f5-85e2-0af12be68b17 -nographic -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-417B0807.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -kernel /var/lib/eucalyptus/instances//admin/i-417B0807/kernel -append root=/dev/sda1 console=ttyS0 -device lsi,id=scsi0,bus=pci.0,addr=0x5 -drive file=/var/lib/eucalyptus/instances//admin/i-417B0807/disk,if=none,id=drive-scsi0-0-0,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -device e1000,vlan=0,id=net0,mac=d0:0d:41:7b:08:07,bus=pci.0,addr=0x4 -net tap,fd=63,vlan=0,name=hostnet0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances//admin/i-417B0807/console.log -device isa-serial,chardev=serial0 -usb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
libvir: Security Labeling error : internal error error calling aa_change_profile()
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name i-417B0807 -uuid 05fd8c59-5169-aea5-740d-328872f50752 -nographic -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-417B0807.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -kernel /var/lib/eucalyptus/instances//admin/i-417B0807/kernel -append root=/dev/sda1 console=ttyS0 -device lsi,id=scsi0,bus=pci.0,addr=0x5 -drive file=/var/lib/eucalyptus/instances//admin/i-417B0807/disk,if=none,id=drive-scsi0-0-0,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -device e1000,vlan=0,id=net0,mac=d0:0d:41:7b:08:07,bus=pci.0,addr=0x4 -net tap,fd=63,vlan=0,name=hostnet0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances//admin/i-417B0807/console.log -device isa-serial,chardev=serial0 -usb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
libvir: Security Labeling error : internal error error calling aa_change_profile()
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name i-417B0807 -uuid 63da946b-99a4-f806-9b17-23327063cafe -nographic -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-417B0807.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -kernel /var/lib/eucalyptus/instances//admin/i-417B0807/kernel -append root=/dev/sda1 console=ttyS0 -device lsi,id=scsi0,bus=pci.0,addr=0x5 -drive file=/var/lib/eucalyptus/instances//admin/i-417B0807/disk,if=none,id=drive-scsi0-0-0,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -device e1000,vlan=0,id=net0,mac=d0:0d:41:7b:08:07,bus=pci.0,addr=0x4 -net tap,fd=63,vlan=0,name=hostnet0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances//admin/i-417B0807/console.log -device isa-serial,chardev=serial0 -usb -device ...

[Dave Walker (davewalker)]

Thanks for reporting this C, with apparmour stopped - can you still reproduce this issue?

[C de-Avillez (hggdh2)]

I did not stop apparmour, but just set it in compain mode:

sudo aa-complain /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd

then started a new instance -- it went to running, and I successfully SSH-ed in:

ubuntu@cempedak:~/uec-testing-scripts/users/admin$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i uectest-k0.priv ubuntu@10.55.55.100
Warning: Permanently added '10.55.55.100' (RSA) to the list of known hosts.
Linux ip-172-19-4-2 2.6.35-10-virtual #15-Ubuntu SMP Thu Jul 22 14:23:36 UTC 2010 i686 GNU/Linux
Ubuntu maverick (development branch)
ubuntu@ip-172-19-4-2:~$

[Thierry Carrez (ttx)]

Tentatively assigning to Jamie, feel free to redistribute

[Dave Walker (davewalker)]

This could be a dupe of bug #599450. A fix is committed for that bug, when that is uploaded we'll get a better idea.

[Jamie Strandboge (jdstrand)]

This is almost certainly a dupe of bug #599450. Please look at that bug or attach the dmesg and mark this as a duplicate if appropriate.

[Thierry Carrez (ttx)]

Fixed now (tested 20100803 ISO), was probably a dupe of 599450