domU fails to get network connection

[Todd Deshane]

I finally got past bug 144631, but some think that network problems may be related to bug 144631.

The domU finally loads, but can't get an IP address via dhcp.

It works fine on the old kernel 2.6.19-4-generic-amd64.

The only errors that I can see are when xend starts/restarts:

ifdown: interface eth0 not configured
SIOCSIFNAME: Device or resource busy

I double-checked xend-config.sxp and it is using
(network-script 'network-bridge')

I will also go back and try the old kernel to sanity check that I didn't mess anything up.

references
https://bugs.launchpad.net/ubuntu/+source/xen-tools/+bug/

[Todd Deshane (deshantm)]

I am on the old kernel and the network connection on the domU works fine. I can dchp an address, ping and SSH to the domU with no problems too.

Something is broken in the latest Xen-3.1 packages.

[André Anjos (andre.anjos-deactivatedaccount-deactivatedaccount)]

I can confirm this one. I have exactly the same thing you report, it happens with gutsy for me: `uname -a` shows
Linux pcuw32 2.6.22-14-xen #1 SMP Tue Oct 9 11:22:33 GMT 2007 i686 GNU/Linux

[Jon Ortuondo (bicyus)]

well, i don't have problems with networking on DomU, but if it can help, i'll post some information.

Kernel AMD64
Linux Silver 2.6.22-14-xen #1 SMP Sun Oct 14 23:20:20 GMT 2007 x86_64 GNU/Linux

Ethernet
00:07.0 Bridge: nVidia Corporation MCP61 Ethernet (rev a2)

Maybe it's a hardware specific bug? Related to the 2.6.22-14-xen kernel?

Hope it helps

[Todd Deshane (deshantm)]

I have two similar ethernet cards:

00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)

So, it doesn't rule out the possibility of hardware-specific. But, it still could be a configuration problem.

Could you post your /etc/xen/xend-config.sxp, guest config, and the type of image and how it was made (i.e. xen-image-create, debootstrap, etc.)

Thanks.

[Paul Waldo (pwaldo)]

I too have this problem on a generic Pentium 4 system with a single Ethernet card. Here are my specifics:
dom0 is gutsy
root@hydra:/etc/xen# dpkg -l "*xen*"|grep ii
ii libc6-xen 2.6.1-1ubuntu9 GNU C Library: Shared libraries [Xen version
ii libxen3.1 3.1.0-0ubuntu18 library interface for Xen, a Virtual Machine
ii linux-image-2.6.22-14-xen 2.6.22-14.46 Linux kernel image for version 2.6.22 on Thi
ii linux-image-xen 2.6.22.14.21 Linux kernel image on Xen
ii linux-restricted-modules-2.6.22-14-xen 2.6.22.4-14.9 Non-free Linux 2.6.22 modules on Xen
ii linux-restricted-modules-xen 2.6.22.14.21 Restricted Linux modules on Xen
ii linux-ubuntu-modules-2.6.22-14-xen 2.6.22-14.37 Ubuntu supplied Linux modules for version 2.
ii linux-xen 2.6.22.14.21 Complete Linux kernel on Xen
ii python-xen-3.1 3.1.0-0ubuntu18 python bindings for Xen, a Virtual Machine M
ii ubuntu-xen-server 0.0.1-2ubuntu4 Xen software for running on servers
ii xen-docs-3.1 3.1.0-0ubuntu18 documentation for XEN, a Virtual Machine Mon
ii xen-hypervisor-3.1 3.1.0-0ubuntu18 The Xen Hypervisor for i386, amd64 amd lpia
ii xen-ioemu-3.1 3.1.0-0ubuntu18 XEN administrative tools
ii xen-tools 3.5-1ubuntu2 Tools to manage debian XEN virtual servers
ii xen-utils-3.1 3.1.0-0ubuntu18 XEN administrative tools
root@hydra:/etc/xen# uname -a
Linux hydra 2.6.22-14-xen #1 SMP Mon Oct 15 00:35:38 GMT 2007 i686 GNU/Linux

domU creation command: xen-create-image --hostname gutsy2

[Paul Waldo (pwaldo)]

Sorry, but this interface allows only one file upload

[Paul Waldo (pwaldo)]

Sorry, but this interface allows only one file upload

[Paul Waldo (pwaldo)]

Sorry, but this interface allows only one file upload

[Fred H (fred-hensley-bereanservices)]

I too am experiencing precisely the same issues as reported above on my Dell 430SC server with dual core pentium D920 processors running 64 bit Ubuntu 7.10 gutsy server and Xen 3.1 ( 2.6.22-14-xen #1 SMP Sun Oct 14 23:20:20 GMT 2007 x86_64 GNU/Linux).

However, immediately after initially creating my two gutsy "guests" on this server via xen-tools they worked flawlessly, including the networking portion. I could ping them, ssh into them, and from within the guest OS use the network interface just fine. Only after shutting down the guest (xm shutdown) and restarting (xm create) did the boot lockup (see bug report 144631) hit them hard. I performed the two referenced work arounds for that bug, but the network connectivity is non-working.

So, bottom line, is there any new progress on this bug? Do these two bugs (144631 and 150805) seem somehow related? Are there any other work-arounds which can help?

Personally, I'm deciding whether to do one or more of the following: (a) downgrade back to xen 3.04, (b) downgrade ubuntu to 7.04 (feisty) or 6.06 (dapper), or (c) simply downgrade the kernel to 2.6.19-4.. Someone posted earlier in the 144631 bug report that the 2.6.19-4 kernel only supported one guest instance, but that did not make sense to me why that would be the case...

Anyone? Anyone? Buehler? :) :)

Thanks,

-Fred-

[Fred H (fred-hensley-bereanservices)]

I went back and followed Todd Deshane's input above, reverting to revert to the earlier kernel (2.6.19-4-generic-amd64) for both my dom0 and domU. However, I am still experiencing the same problems as reported above with the latest kernel. After using "xen-create-image" to create the image and "xm create" the first time, the guest is created and networking works just fine..

However, if I do an "xm shutdown" on the newly created guest, followed by another "xm create" using the same guest configuration file, the same guest created again cannot access the network interfaces.. When attempting to do a simple ping, I get the following response:

"connect: Network is unreachable"

As the next step, I went back and reviewed the /var/log/xen log file generated when creating the xen image, and noticed the following lines of interest:

Running hook 40-setup-networking

/usr/lib/xen-tools/gutsy.d/40-setup-networking: 139: bcast: not found

hook 40-setup-networking: done.

Why does the networking error occur in the log file when generating the xen image? Why does guest correctly access the network the first time it is created via "xm create", but not after it is shutdown (via "xm shutdown") and subsequently recreated again via "xm create"?

Is this a xen 3.1 problem, a xen-tools 3.1 problem, or perhaps both?

Finally, in the interests of full disclosure, I have set the following in my xend-config.sxp:

(network-script network-bridge)
(vif-script vif-bridge)

Please advise if I can assist in any further testing of this issue, or to provide any other supporting detail in helping resolve this problem.

Thanks,

-Fred-

[Todd Deshane (deshantm)]

Fred: So, you can look can look closer at networking as you test this.

Run a network analyzer such as wireshark, tshark, or similar on the dom0 when you ping the dom0. See if anything is going on. You should also be able to trace the virtual network card associated with the guest.

Other things to try:

use strace on the domU when running some of the simple ping or other network commands.

run commands like route, and ifconfig -a, and dmesg, and lspci, and lsmod, insmod <network module> on the guest to try to get some more intuition on that what it happening.

Also look in more log files both on the guest and on dom0 /var/log/messages, /var/log/syslog, /var/log/xen/*

Try reloading networking in the domU and watch dom0 log files.

To me it doesn't make sense that networking works when it first boots and then doesn't on the second boot.

Also, we really need to push on the latest kernels and packages, since those will be the ones that will be fixed, it is unlikely anybody will take the time to fix any of the old ones.

I am soon going to focus my efforts on whatever is released for hardy, maybe that can get back ported.

[Fred H (fred-hensley-bereanservices)]

It will be a few days before I can get to this, but will be happy to provide this data as soon as possible..

Thanks for the all of the helpful feedback on what to look for.

Regards,

Fred

Best Regards,

Fred Hensley
President
Berean IT Services
1115 Mill Springs
Richardson, Texas 75080
214.725.5434 (V)
214.206.1957 (F)
<email address hidden>
http://www.bereanservices.com

-----Original Message-----
From: Todd Deshane <email address hidden>

Date: Fri, 16 Nov 2007 13:20:09
To:<email address hidden>
Subject: [Bug 150805] Re: domU fails to get network connection

Fred: So, you can look can look closer at networking as you test this.

Run a network analyzer such as wireshark, tshark, or similar on the dom0
when you ping the dom0. See if anything is going on. You should also be
able to trace the virtual network card associated with the guest.

Other things to try:

use strace on the domU when running some of the simple ping or other
network commands.

run commands like route, and ifconfig -a, and dmesg, and lspci, and
lsmod, insmod <network module> on the guest to try to get some more
intuition on that what it happening.

Also look in more log files both on the guest and on dom0
/var/log/messages, /var/log/syslog, /var/log/xen/*

Try reloading networking in the domU and watch dom0 log files.

To me it doesn't make sense that networking works when it first boots
and then doesn't on the second boot.

Also, we really need to push on the latest kernels and packages, since those will be the ones that will be fixed, it is unlikely anybody will take the time to fix any of the old ones.

I am soon going to focus my efforts on whatever is released for hardy,
maybe that can get back ported.

--
domU fails to get network connection
https://bugs.launchpad.net/bugs/150805
You received this bug notification because you are a direct subscriber
of the bug.

[Eadz (eadz)]

I have exactly the same problem as Fred H, and it is very strange.

Using a fresh gutsy install, installed ubuntu-xen-server, and domUs work after first creating then booting them but then networking fails trying to boot the domU a second time with eth0: error fetching interface information: Device not found

This is on a HP DL360G3 dual intel xeon 2.8ghz.

[Lilith Boids (lilith-boids)]

Same story here.

After getting past the 'Mount Filesystem'-Bug and the 'HW Clock'-Bug I was beaten by no networking. I double checked every step I made, reinstalled everything, changed the hardware to a real machine and finally found the missing eth0.

It isn't eth0 anymore. If I take a look at /proc/net/dev_snmp6, i get my 'missing' interafce.
It started with eth4, and increased it's number every time I made a shutdown / create. However, if I change the interface in /etc/networking to what i found in dev_snmp_6 and restart networking, it's working again.

'till I restart the DomU again. Then it increases once more. Actually I'm on eth11: atm...

[Nat Blundell (nat-isotoma)]

I had the same problem with Gutsy domU on Gutsy dom0. It is caused by xend generating a new MAC address each boot, udev recognises this as a new card and therefore creates a new entry.

Add MAC address to the domU config file like:

vif = [ 'mac=00:11:22:33:44:55' ]

See http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/user.html#SECTION02220000000000000000 for more info on what MAC address you should use.

[Eadz (eadz)]

can anyone confirm this workaround? I've switched to debian because of this bug but if this workaround works i'll switch back..

[Todd Deshane (deshantm)]

The setting of the mac address, does fix the udev problem, but I don't seem to have networking working yet though.

Is there any other steps that are needed?

[Todd Deshane (deshantm)]

so I looked into it some more.

I have a work around that I can live with (for now). Better solutions and working on the box are going to be for hardy hopefully.

I noticed that I got the following errors in /etc/xen/xend-debug.log:
ifdown: interface eth0 not configured
SIOCSIFNAME: Device or resource busy

So, I stopped xend and tried /etc/xen/scripts/network-bridge star manually and the same errors appeared...:
#/etc/xen/scripts/network-bridge start
ifdown: interface eth0 not configured
SIOCSIFNAME: Device or resource busy

So, I remove network-manager and added the following to /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp

I now I have networking in my guest.

(Also, don't forget to restart xend)

I rebooted the guest and networking still works by the way.

Another thing to note is that if you have multiple network cards xen can't always automatically figure out which one to use as the bridge.

So, if you don't need the network card you can take it out. Or carefully specify in /etc/xen/xend-config.sxp. Or if you just don't want a card to show up you can hide it from dom0. In this case it can actually be used by a guest directly too...

To hide it add the pciback stuff to the dom0 kernel command line, an example follows:

module /boot/vmlinuz-2.6.22-14-xen root=UUID=6af9903c-e2ee-4fe3-86e2-3601d97131a2 ro console=tty0 pciback.permissive pciback.hide=(03:00.0)

If you then want a guest to be able to use that card instead, you can then use something like

pci='03:00.0'

in the guest config file.

I use 03:00.0 since that is the card i wanted to hide. I determined this with the lspci command as below.

# lspci | grep net
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02)
04:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)

More testing will be needed, but at least it is in a working state.

Hope that helps.

[Frederik Carlier (frederik-carlier)]

I experienced the same problem, Todd Deshane's workaround (add an interface eth0 to /etc/network/interfaces) solved the problem for me.

[Will Saxon (saxonww)]

I had an eth0 created normally the first time I started a gutsy domU, but repeated restarts generated additional eth* devices (I am on eth13). I am not very familiar with network-manager or the difference between -workstation and -server, but I don't appear to be using network-manager as I already had an eth0 entry in /etc/network/interfaces. So the problem for me is that this entry now points to an interface which doesn't exist.

I think the best way to work around this is to specify the mac address in the domain.cfg configuration as pointed out above. But how do you determine what mac address was assigned to the 'original' eth0? An alternative would be to cause udev to forget previous known mac addresses, but I don't know how to do that either.

[Will Saxon (saxonww)]

OK so, every time it increments the interface number, it stores the new entry in /etc/udev/rules.d/70-persistent-net.rules. My 'original' eth0 had a mac address of 00:16:3e:61:a6:be.

So all I needed to do was specify the desired MAC address in my domain.cfg:

#
# Networking
#
vif = [ 'ip=192.168.1.20,mac=00:16:3e:61:a6:be' ]

And that fixed the problem. Now when I restart the gutsy domU, I get eth0 autoconfigured and networking works as expected.

[William Grant (wgrant)]

Alternatively, open up /etc/udev/rules.d/70-persistent-net.rules, and remove the ATTRS{address} parameter entirely. That stops udev using a different name for each MAC.

[Andreas Gustafsson (gson)]

I also ran into this problem, on 7.10 (desktop). I already had the mac address
set in the guest configuration file, but networking didn't work in the guest.
I finally got it to work by applying Todd Deshane's workaround of removing
network-manager and adding

  auto eth0
  iface eth0 inet dhcp

to /etc/network/interfaces.

[agent 8131 (agent-8131)]

There are 2 behaviors that seem to be part of this report.

1. By default Xen will assign a random mac address and by default Ubuntu will assign each new mac address to a new interface. I don't expect either behavior to change and suggest that it is wise to assign a static address to each DomU anyway. Though I wouldn't be opposed to Xen having a different default behavior.

2. Having Network Manager installed on a Dom0 seems to cause some issues. I experienced this today on a client's computer where, to the best I could determine, Network Manager will screw up the networking in the Dom0 by getting an ip for peth0. And once Dom0 was using a static ip I found that Network Manager would screw up the networking for the DomU as well. I think it best that servers not have Network Manager installed. If someone uses Xen on a desktop and wants Network Manager than this will have to be explored in more detail.

[rusivi2 (rusivi2-deactivatedaccount)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. The issue that you reported should be reproducible with the live environment of the Desktop CD development release - Maverick Meerkat. It would help us greatly if you could test with it so we can work on getting it fixed in the next release of Ubuntu. You can find out more about the development release at http://www.ubuntu.com/testing/. Thanks again and we appreciate your help.

[Thomas Hotz (thotz-deactivatedaccount)]

Can you please try to reproduce this with a supported Ubuntu version? Thank you!

[Launchpad Janitor (janitor)]

[Expired for xen-3.2 (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for xen-3.1 (Ubuntu) because there has been no activity for 60 days.]