tun module no longer automatically available (was: Euca 1.6.2 fails to boot an instance)

[Thierry Carrez]

Testing 1.6.2~bzr1103-0ubuntu1
The node controller can't boot an instance. The relevant code snippet is:

[EUCAINFO ] currently running/booting: i-465307DE
[EUCAERROR ] libvirt: Failed to add tap interface to bridge 'br0': No such file or directory (code=38)
[EUCAFATAL ] hypervisor failed to start domain

[Thierry Carrez (ttx)]

This does not occur on upgraded systems. I'll test again with the bzr1113 daily.

[Thierry Carrez (ttx)]

Marking as invalid for eucalyptus upstream, since it does appear to be some env setup issue.

[Thierry Carrez (ttx)]

Some help from upstream:
"The log message indicates that either the interface 'br0' does not exist on your node (i.e. the node has not yet been configured with a bridge named 'br0' as it's primary interface), or that libvirt/kvm is unable to attach the tap device that KVM creates upon VM start to that bridge (this can sometimes be caused by apparmor getting in the way). I believe the right tactic is to first check for bridge existence, then start inspecting apparmor/libvirt/kvm issues."

[Thierry Carrez (ttx)]

Alpha1 install 1.6.1: PASS
Upgrade eucalyptus to bzr1103: PASS
Upgrade remaining system to current: PASS
Reinstall with latest daily (bzr1103): FAIL

Must be some one-time environment setup that was done in the past, I'll dig more.

[Thierry Carrez (ttx)]

Bridge br0 exists, apparmor doesn't get in the way... it's coming from something else.

New test:
Alpha1 install 1.6.1: PASS
Upgrade remaining system to current, reboot: PASS
Dist-upgrade system to current, reboot: FAIL

The dist-upgrade picks up the following updates / new packages:
initramfs-tools 0.92bubuntu55 -> 0.92bubuntu62
initramfs-tools-bin NEW -> 0.92bubuntu62
libplymouth2 NEW -> 0.8.0~-6
libpng12-0 NEW -> 1.2.41-1
linux-headers-server 2.6.32.7.7 -> 2.6.32.9.9
linux-headers-2.6.32-9 NEW -> 2.6.32-9.13
linux-headers-2.6.32-9-server NEW -> 2.6.32-9.13
linux-image-server 2.6.32.7.7 -> 2.6.32.9.9
linux-server 2.6.32.7.7 -> 2.6.32.9.9
linux-image-2.6.32-9-server NEW -> 2.6.32-9.13
mountall 1.1 -> 2.2

Rebooting, selecting 2.6.32-7 -> still FAIL
Reverting initramfs/mountall update, rebooting on 2.6.32-9 -> PASS

So this is apparently coming from the recent mountall/initramfs-tools update, though I haven't time left to investigate more *why*. Adding a mountall task to track this.

[Kees Cook (kees)]

I've hit this same issue with all my libvirt-managed kvm VMs too:

$ virsh start sec-hardy-i386
error: Failed to start domain sec-hardy-i386
error: Failed to add tap interface to bridge 'br0': No such file or directory

[Kees Cook (kees)]

Ah-ha! I suspect it's recent changes to the initramfs that continues to reduce the modules-needed-for-boot, which left out "tun". I was able to solve this by just "sudo modprobe tun" before attempting to start an interface. Hint found at:

http://www.gasid.org.uk/2009/02/kvm-virt-install-failed-to-add-tap-interface/

[Thierry Carrez (ttx)]

@Kees: Yes, that makes sense. Same issue seems to hit openvpn, bug 502081.

[Dustin Kirkland  (kirkland)]

Per IRC conversation, the Kernel team will probably just build tun into the kernel.

Marking invalid against the rest of the affected packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.32-10.14

---------------
linux (2.6.32-10.14) lucid; urgency=low

  [ Alex Deucher ]

  * SAUCE: drm/radeon/kms: fix LVDS setup on r4xx
    - LP: #493795

  [ Andy Whitcroft ]

  * Revert "(pre-stable) acpi: Use the ARB_DISABLE for the CPU which model
    id is less than 0x0f."
  * config-check -- ensure the checks get run at build time
  * config-check -- check the processed config during updateconfigs
  * config-check -- CONFIG_SECCOMP may not be present
  * TUN is now built in ignore
  * SAUCE: acpi battery -- delay first lookup of the battery until first
    use
  * SAUCE: async_populate_rootfs: move rootfs init earlier
  * ubuntu: AppArmor -- update to mainline 2010-01-06
  * SAUCE: move RLIMIT_CORE pipe dumper marker to 1
    - LP: #498525

  [ Dave Airlie ]

  * (pre-stable) drm/radeon/kms: fix crtc vblank update for r600

  [ Leann Ogasawara ]

  * Add asix to nic-usb-modules file
    - LP: #499785

  [ Peter Zijlstra ]

  * (pre-stable) sched: Fix balance vs hotplug race

  [ Tim Gardner ]

  * [Config] Enable CONFIG_FUNCTION_TRACER
    - LP: #497989
  * [Config] Drop lpia from getabis
  * [Config] Build in TUN/TAP driver
    - LP: #499491
  * [Config] DH_COMPAT=5

  [ Upstream Kernel Changes ]

  * Revert "(pre-stable) drm/i915: Avoid NULL dereference with
    component_only tv_modes"
  * Revert "(pre-stable) drm/i915: Fix sync to vblank when VGA output is
    turned off"
  * USB: usb-storage: fix bug in fill_inquiry
  * USB: option: add pid for ZTE
  * firewire: ohci: handle receive packets with a data length of zero
  * rcu: Prepare for synchronization fixes: clean up for non-NO_HZ handling
    of ->completed counter
  * rcu: Fix synchronization for rcu_process_gp_end() uses of ->completed
    counter
  * rcu: Fix note_new_gpnum() uses of ->gpnum
  * rcu: Remove inline from forward-referenced functions
  * perf_event: Fix invalid type in ioctl definition
  * perf_event: Initialize data.period in perf_swevent_hrtimer()
  * perf: Don't free perf_mmap_data until work has been done
  * PM / Runtime: Fix lockdep warning in __pm_runtime_set_status()
  * sched: Check for an idle shared cache in select_task_rq_fair()
  * sched: Fix affinity logic in select_task_rq_fair()
  * sched: Rate-limit newidle
  * sched: Fix and clean up rate-limit newidle code
  * x86/amd-iommu: attach devices to pre-allocated domains early
  * x86/amd-iommu: un__init iommu_setup_msi
  * x86, Calgary IOMMU quirk: Find nearest matching Calgary while walking
    up the PCI tree
  * x86: Fix iommu=nodac parameter handling
  * x86: GART: pci-gart_64.c: Use correct length in strncmp
  * x86: ASUS P4S800 reboot=bios quirk
    - LP: #366682
  * x86, apic: Enable lapic nmi watchdog on AMD Family 11h
  * ssb: Fix range check in sprom write
  * ath5k: allow setting txpower to 0
  * ath5k: enable EEPROM checksum check
  * hrtimer: Fix /proc/timer_list regression
  * ALSA: hrtimer - Fix lock-up
  * ALSA: hda - Terradici HDA controllers does not support 64-bit mode
  * KVM: x86 emulator: limit instructions to 15 bytes
  * KVM: s390: Fix prefix register checking in arch/s390/kvm/sigp.c
  * KVM: s390: Make psw available on all exits...

[whoami (prchauha)]

tun module does not seem to be available in Kernel 2.6.32-17

* Tried using modconf, but was unable to find "tun" under kernel/drivers/net
* sudo module-assistant search tun returns
tun-source (source package not installed):
  -- Binary package(s) for kernel(s):
   + (2.6.32-17-generic):
* sudo module-assistant a-i tun-source (Fails)
* sudo modprobe tun (Does not have any effect)
* Editing /etc/module file and adding "tun" to it does not work
* lsmod | grep tun (Does not show anything)
* ls -l /dev/net/tun show output "/dev/net/tun"
* ifconfig -a does not show any tun0 interface
* Finally VPNC / KVPNC are failing to work.

Any help in this regards would be appreciated, as I've tried hard but still issue remains.

P.S. Please don't strip the Kernel for faster boot to such an extent that there is nothing left to boot.

[Dustin Kirkland  (kirkland)]

* ls -l /dev/net/tun show output "/dev/net/tun"

This is because tun is now built into the kernel. It's most certainly there, present in -17.

Please do not re-open this bug against Eucalyptus.

[Mathieu Trudel-Lapierre (cyphermox)]

Closing vpnc/kvpnc/network-manager tasks. It's not an issue in network-manager or vpnc; they were just inconvenienced by the fact that the module wasn't available.

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lucid' to 'verification-done-lucid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!