linking two virtual machines does not work

[Thomas Schweikle]

Binary package hint: bridge-utils

Configured bridge:

auto br8
iface br8 inet static
  address 172.16.28.1
  netmask 255.255.255.0
   bridge_fd 9
  bridge_hello 2
  bridge_maxage 12
  bridge_stp on
  bridge_ports eth0.6
  post-up iptables -t nat -A POSTROUTING -s 172.16.28.0/24 ! -d 172.16.28.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
  post-up iptables -t nat -A POSTROUTING -s 172.16.28.0/24 ! -d 172.16.28.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
  post-up iptables -t nat -A POSTROUTING -s 172.16.28.0/24 ! -d 172.16.28.0/24 -j MASQUERADE
  post-up iptables -t filter -A FORWARD -d 172.16.28.0/24 -o br8 -m state --state RELATED,ESTABLISHED -j ACCEPT
  post-up iptables -t filter -A FORWARD -s 172.16.28.0/24 -i br8 -j ACCEPT
  post-up iptables -t filter -A FORWARD -i br8 -o br8 -j ACCEPT
  post-up iptables -t filter -A FORWARD -o br8 -j REJECT --reject-with icmp-port-unreachable
  post-up iptables -t filter -A FORWARD -i br8 -j REJECT --reject-with icmp-port-unreachable

Interface is configured as expected:
br8 Link encap:Ethernet Hardware Adresse 00:24:1d:df:f0:38
          inet Adresse:172.16.28.1 Bcast:172.16.28.255 Maske:255.255.255.0
          inet6-Adresse: fe80::224:1dff:fedf:f038/64 Gültigkeitsbereich:Verbindung
          inet6-Adresse: 172.16.28.1/96 Gültigkeitsbereich:Global
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metrik:1
          RX packets:1990 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:0
          RX bytes:365337 (365.3 KB) TX bytes:706 (706.0 B)

iptables are configured as expected:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 172.16.28.0/24 state RELATED,ESTABLISHED
ACCEPT all -- 172.16.28.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

This bridge connects two kvm-systems:
br8 8000.00241ddff038 yes eth0.6
                                                        vnet2
                                                        vnet9

System one:
eth4 Link encap:Ethernet HWaddr 52:54:00:ae:6c:73
          inet addr:172.16.28.25 Bcast:172.16.28.255 Mask:255.255.255.0
          inet6 addr: 172.16.28.25/96 Scope:Global
          inet6 addr: fe80::5054:ff:feae:6c73/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:863 errors:0 dropped:0 overruns:0 frame:0
          TX packets:107 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:69740 (69.7 KB) TX bytes:13207 (13.2 KB)
          Interrupt:11 Base address:0x6200

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.16.28.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4

On this system a dhcp-server is active. The server listens on port 67 on all open interfaces.

System two:
eth0 Link encap:Ethernet HWaddr 52:54:00:ae:6c:73
          inet6 addr: fe80::5054:ff:feae:6c73/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0 KB) TX bytes:0 (0 KB)
          Interrupt:11 Base address:0x6200

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface

Command "dhclient eth0" just times out:
Listening on LPF/eth0/52:54:00:05:cf:85
Sending on LPF/eth0/52:54:00:05:cf:85
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 18
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
No DHCPOFFERS received.
No working leases in persistent database - sleeping

No dhcp-packets are seen by the client, nor the server. Impossible to use dhcp with kvm.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: bridge-utils 1.4-5ubuntu2
ProcVersionSignature: Ubuntu 2.6.35-23.41-server 2.6.35.7
Uname: Linux 2.6.35-23-server x86_64
Architecture: amd64
Date: Mon Dec 20 22:31:35 2010
InstallationMedia: Ubuntu-Server 10.10 "Maverick Meerkat" - Release amd64 (20101007)
ProcEnviron:
 LC_CTYPE=en_US.UTF-8
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: bridge-utils

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug and helping to make Ubuntu better.

As an experiment, please try the following. Create a simple bridge,

    brctl addbr br99

and edit the VM configurations to put them on br99 instead of br8. (Are you using libvirt, or starting them by hand. Please document how you configured them) Does the server now get the dhcp requests?

[Thomas Schweikle (tps)]

I've switched to use "br180" for the bridge. No change:

brctl addbr br180
brctl addif br180 eth1
ifconfig br180 172.16.180.1/24

VM starts using this bridge, but does not receive any broadcast. DHCP does not work.

[Thomas Schweikle (tps)]

I've changed the VM using "virsh edit".

[Serge Hallyn (serge-hallyn)]

Haven't yet been able to reproduce (so far on natty, though, not maverick).
How I tried:

1. created new 'priv' network in libvirt:
 cd /etc/libvirt/qemu/networks
 cat > priv.xml << EOF
 <network>
   <name>priv</name>
   <bridge name='virbr%d' stp='on' delay='0' />
   <ip address='192.168.123.1' netmask='255.255.255.0'>
   </ip>
 </network>
 EOF
 virsh net-create priv.xml
2. restart libvirt
3. create two virtual machines, one a dhcp server, one a dhcp client
4. set up the dhcp server:
 cat >> /etc/dhcp3/dhcpd.conf << EOF
 subnet 192.168.123.0 netmask 255.255.255.0 {
   range 192.168.123.20 192.168.123.20;
 }
 EOF
 /etc/init.d/dhcp-server start

Then, running dhclient3 on the client VM gets an ip address from
the dhcp server.

I'll try to reproduce these same steps using plain qemu on a
maverick VM.

[Serge Hallyn (serge-hallyn)]

Actually, I'm not sure how soon I can get a maverick host up with space for two guests. Thomas, could you retry my steps above precisely and check whether those succeed?

I will however try to reproduce with two network namespaces in a maverick instance.

[Serge Hallyn (serge-hallyn)]

On an uptodate maverick system, I created two network namespaces,
one as dhcp server, the other as client. The client was able to
get an address from the server. What I did:

debootstrap lucid b1
debootstrap lucid b2
chroot b1
 mount -t proc proc /proc
 apt-get install vi dialog dhcp3-server
 umount /proc
 exit
chroot b2
 mount -t proc proc /proc
 apt-get install vi dialog dhcp3-client
 umount /proc
 exit
brctl addbr br0
# get ns_exec
apt-get install git gcc
git clone git://git.sr71.net/~hallyn/cr_tests.git
cd cr_tests
git checkout ns_exec
make ns_exec
cp ns_exec /bin/

# set up the networking
ip link add type veth
ip link add type veth
ifconfig veth0 up
ifconfig veth1 up
ifconfig veth2 up
ifconfig veth3 up
brctl addif br0 veth0
brctl addif br0 veth2

# create the containers
cd b1 # do this in a new terminal
 ns_exec -cmnp /bin/bash
 chroot .
 mount -t proc proc /proc
 mount -t sysfs sys /sys
 vi /etc/dhcp3/dhcpd.conf # add the same lines as in the previous comment
 # leave it running

cd b2 # do this in a new terminal
 ns_exec -cmnp /bin/bash
 chroot .
 mount -t proc proc /proc
 mount -t sysfs sys /sys
 # leave it running

# pass the interfaces to the containers.
# Find the pids of the bash shells parented by the ns_exec processes,
# i.e. 'ps -ef' will show two ns_exec's, each with a bash process following,
# call the first pid $p1 and the next $p2

ip link set veth1 netns $p1
ip link set veth3 netns $p2

# back in container b1
 ifconfig veth1 192.168.254.10 up
 /etc/init.d/dhcp3-server start

# in container b2
 dhclient

Container b2 gets an ip address served by b1.

So the kernel and bridge-utils packages in maverick appear to be fine.
Something else is going on on Thomas' system.

[Thomas Schweikle (tps)]

Tried the same. OK if done on *ONE* system. I can ping every vm running on this Server.

Now back to my setup.

First system:
ifconfig eth0.5 0.0.0.0 up
brctl addbr vm5
brctl addif vm5 eth0.5
ifconfig vm5 192.168.1.1/24 up

Second system:
ifconfig eth0.5 0.0.0.0 up
brctl addbr vm5
brctl addif vm5 eth0.5
ifconfig vm5 192.168.1.2/24 up

On first system:
ping 192.168.1.1
-> OK. Works
ping 192.168.1.2
-> no output. Only lost pings

On second system:
ping 192.168.1.2
-> OK. Works.
ping 192.168.1.1
-> no output. Only lost pings

Both systems are connected.

Adding VMs to the vm5-bridge on first system all guests can ping each other.
Adding VMs to the vm5-bridge on second system all guests can ping each other.

if a guest on first system tries to ping one on second system, the guest on second system is unreachable.
If a guest on second system tries to ping one on first system, the guest on first system is unreachable.

[Serge Hallyn (serge-hallyn)]

Hi Thomas,

exactly how are the systems connected? How are eth0 and eth0.5 configured?

If you simply

First system:
ifconfig eth0.5 192.168.1.1/24 up

Second system:
ifconfig eth0.5 192.168.1.1/24 up

are they able to ping each other?

How about if you use eth0 rather than eth0.5 in the bridge?

[Santiago Garcia Mantinan (manty)]

I'm afraid the only problem here is that both machines have the same physical address, thus there is no bug on the bridge or bridge-utils, just the typical problems you would find in having two machines with the same MAC address.

Hope that helps.

Regards.

[Serge Hallyn (serge-hallyn)]

@Santiago - thanks very much for noticing that.

[Thomas Schweikle (tps)]

No, this is not the case. Both machines are assigned network unique MAC-Addresses. I've checked them:

1: 52:54:00:9a:a1:18
2: 52:54:00:9a:a1:20

[Serge Hallyn (serge-hallyn)]

@Thomas

which NICs have the MAC addrs listed in comment #13?

Note that the description shows both eth4 on the dhcp server VM
and eth0 on the dhcp client VM has having MAC address
52:54:00:ae:6c:73. That is what Santiago was pointing out.

[Thomas Schweikle (tps)]

These are the mac addresses for eth4 and eth0 respectively.
Looks like I've taken a listing from before I changed mac addresses. :-(