Ubuntu

ease cloning of virtual images by disabling mac address rules

Reported by Nikolaus Filus on 2009-03-11
54
This bug affects 7 people
Affects Status Importance Assigned to Milestone
udev (Ubuntu)
Wishlist
Martin Pitt

Bug Description

Cloning an vmware image of ubuntu or ubuntu-based spin-offs like JeOS leads to unwanted side-effects after changing the mac-address as described in
http://communities.vmware.com//thread/198298?tstart=0
and
http://professionalvmware.com/2009/03/10/ubuntu-cloning-mac-address-change-mayhem/
Both sources advise to add the following line to ‘persistent-net-generator.rules’:
  # ignore VMware virtual interfaces
  ATTR{address}=="00:0c:29:*", GOTO="persistent_net_generator_end"

Please apply this to the packages.

What are these devices?

Are they the MAC address of the emulated vmware interface, or the MAC address of the host-side interface?

Changed in udev:
importance: Undecided → Wishlist
status: New → Incomplete
Nikolaus Filus (nfilus) wrote :

Hi,
this is for the emulated NICs inside the image - in this case vmware, but AFAIK it works in the same way for xen, virtualbox and maybe kvm.

Please supply the requested information

Changed in udev (Ubuntu):
status: Incomplete → Invalid

Attaching "ubuntu-bug udev" from a vmware virtual machine.

I'm not sure if the virtual nics appear to the OS as any different from the actual hardware they are emulating, but I do know all virtual NICs inside vmware vms have MAC addresses starting with 00:0c:29 or 00:50:56.

I think everyone who uses Debian or Ubuntu inside a vmware vm experiences this issue when cloning vms or when the MAC address changes for another reason. udev remembers the old mac address and assigns a new interface name, breaking the network configuration of the machine.

Changed in udev (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) wrote :

http://coffer.com/mac_find/?string=vmware confirms these two vendor prefixes from VMWare (and two more).

However, I still don't quite understand why merely blacklisting that vendor will help/won't break anything. What about VMs with more than one NIC? Such as a virtualized firewall? They certainly need persistent ethN enumeration just as much as "real" machines with multiple NICs?

Changed in udev (Ubuntu):
status: Invalid → Incomplete
Martin Pitt (pitti) wrote :

Likewise, if a cloned instance gets a new MAC, then the generated rule in /etc/udev/rules.d/70-persistent-net.rules won't match, and thus should be equivalent to not existing at all.

I'm trying to understand what breaks here, and why the blacklisting would help with that.

I hadn't considered vms with multiple nics and having them enumerated correctly might be worth more than fixing this issue.

On an existing vm with one interface, /etc/udev/rules.d/70-persistent-net.rules might be something like:

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM ... ATTR{address}=="00:50:56:80:68:23" ... NAME="eth0"

After a clone the file still has the previous line but also an additional line:

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM ... ATTR{address}=="00:50:56:80:13:f7" ... NAME="eth1"

So the result is there is still only one interface and it's eth1 but the rest of the networking configuration is looking for eth0.

Martin Pitt (pitti) wrote :

Thanks for the explanation, Matthew.

Seems we are between a rock and a hard place here -- we should not break setups with multiple interfaces, but this is indeed an inconvenience, too. The current udev rules already ignores "xen" devices, that would speak for ignoring VMWare as well for consistency.

Unfortunately we can't cleanly do something like "If the machine only has one NIC, do not generate a persistency file", since they might be added/removed/changed any time.

Server team, do you have some input here? (Also wrt. KVM)

Is there any way that udev could see that although the mac address has changed, the device (PCI device 0x8086:0x100f (e1000)) is the same, so modify the existing rule instead of creating a new one?

Martin Pitt (pitti) on 2010-02-05
Changed in udev (Ubuntu):
assignee: Martin Pitt (pitti) → nobody
Scott Moser (smoser) wrote :

The bug here is really no different than if you have a "real" server, and need to change the network card. You'd stop the server, replace the card, start the server and no longer have eth0.

There are a number of things that really *should* be done when "cleaning" an image, this is just one of them. Other things might be removing /root/.bash_history (its an example, possibly not a good one).

I think at the moment this is best left as a "cleaning" step that must occur. Things like this are why I recommend using the pristine tarball rather than a live image when rebundling an ec2 image.

Scott Moser (smoser) wrote :

I just re-read the above and see that xen devices are currently ignored. Given that, I do think it makes sense to also handle vmware (and possibly even virtio devices the same way).

That said, we don't want xen, vmware, or any systems with multiple network interfaces to give non-deterministic ordering. If its not already common, I would expect virtual machines with multiple interfaces to be common in the future. It would be nice to something to handle that correctly.

Would it be possible to do add persisitent rules based not on MAC addr, but on bus location? The idea would be that in a new VM, the devices would be attached the same, but different MACs. if the rule said:
  "device on pci bus at 0000:00:19.0 is eth0"
  "device on pci bus at 0000:00:19.1 is eth1"
That might be persistent across VMs (or at very least could be made to be), and would be deterministic for multiple network devices.

Tom Ellis (tellis) wrote :

Just confirming that this also occurs on a KVM virtual machine, in my case I was using 'virt-clone' to clone a system, which changes the MAC addr... on boot of the clone you have eth0 moved to eth1 due to the persistent udev rule entries.

Martin Pitt (pitti) wrote :

PCI based addressing ought to work, although this would break the (admittedly rather theoretical) possibility to move them to a different slot, etc. For hotpluggable network cards (USB etc, as you might find them on e. g. the Beagle board or other embedded controllers) we must stay at MAC-based identification.

Michael Jones (jonesmz) wrote :

Just my two cents here:

   I ran into this issue last week. I was attempting to clone a virtual machine on a remote server to create a new VM. The lag between my location and the server is big enough to make doing installations remotely a pain, but the commandline clone functionality is superb. When I cloned the template (which is so named because it is the template for new virtualmachines on this server), and booted the new vm, there was no network connectivity. I logged into the machine through the Virtual Machine Manager utility and removed the udev rules for the network card, but it was still a good half hour searching around to find out why it didn't work by itself.

    Thanks for making such good tools, though.

On Thu, 11 Feb 2010, Martin Pitt wrote:

> PCI based addressing ought to work, although this would break the
> (admittedly rather theoretical) possibility to move them to a different
> slot, etc. For hotpluggable network cards (USB etc, as you might find
> them on e. g. the Beagle board or other embedded controllers) we must
> stay at MAC-based identification.

Ok, so assuming "addressed based persistent naming" is desirable, I have
2 thoughts on how it could enabled:
a.) for all cards on the pci bus, use pci based persistent naming.
  This fails in the moving a card from slot 1 to slot 2 case as you
  mentioned above.
b.) for all devices deemed "virtual" use address based persistent naming.
  This fails if we can't determine something to be "virtual". If we
selectively enable for known MAC addrs OR known drivers, then we'd miss
anywhere where the hypervisor implements a emulated device (pcnet32 or
e1000 in kvm) and selects a MAC address that we don't whitelist.

b, seems safer as we can have more strict control on the white list.

Thoughts?

Scott Moser (smoser) wrote :

http://hg.fedorahosted.org/hg/python-virtinst/file/ae6ca033e8c5/virtinst/util.py#l181 indicates that:
- 00:16:3E allocated to xensource
- 52:54:00 used by qemu/kvm

http://libvirt.org/drvesx.html indicates:
- 00:0c:29 and 00:50:56 are owned by vmware

Further, I know that
- d0-0d used by Eucalyptus

maybe it is sufficient to simply apply to these MAC addrs what we already do to xen. It just seems that then we are vulnerable to device/driver ordering issues.

One other thing to consider is that kvm implements pci hotplug for network devices (http://www.linux-kvm.org/page/Hotadd_pci_devices). So whitelisting of kvm addresses would result in inconsistent naming with 2 network devices being added and removed in somewhat arbitrary order.

Scott Moser (smoser) wrote :

I moved this out of incomplete and into triaged. I think everyone here understands the nature of the problem.

Changed in udev (Ubuntu):
status: Incomplete → Triaged

FWIW Debian has ignored VMware virtual interfaces by mac:

If you grab http://ftp.us.debian.org/debian/pool/main/u/udev/udev_157-1_i386.deb and look at lib/udev/rules.d/70-persistent-net-generator.rules it has:

# ignore interfaces with locally administered or null MAC addresses
# and VMWare virtual interfaces
ENV{MATCHADDR}=="?[2367abef]:*", ENV{MATCHADDR}=""
ENV{MATCHADDR}=="00:00:00:00:00:00", ENV{MATCHADDR}=""
ENV{MATCHADDR}=="00:0c:29:*|00:50:56:*", ENV{MATCHADDR}=""

Martin Pitt (pitti) wrote :

This has recently been discussed in #udev, and we found that it does not hurt that much to blacklist them. The kernel reliably sorts eth* enumeration by bus number, so as long as you only have cards from one vendor (or more precisely, drivers), the enumeration will be stable. Having cards from different vendors is very unlikely in VMs.

Marco submitted the blacklisting for kvm:

http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=af29bf0966994bc233282148f519e12ea29e1c5a

Changed in udev (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: Triaged → In Progress
Martin Pitt (pitti) wrote :

udev (162-1) maverick; urgency=low

  * New upstream release. Changes since our previous git snapshot:
    - cdrom_id: Fix DVD-RW media and blank DVD detection.
    - Do not create persistent name rules for kvm/qemu/vmware interfaces.
      (LP: #341006)
  * Add debian/watch.

 -- Martin Pitt <email address hidden> Sun, 12 Sep 2010 20:12:37 +0200

Changed in udev (Ubuntu):
status: Fix Committed → Fix Released
Richard Thomas (7-sd-b) wrote :

While it helps in a pragmatic kind of way, carving out exemptions is really a kludge.

The issue is that both aspects of the issue are valid. Persistence is important in many situations but in others, it's more important to not have things changing when they don't need to be changing. Arguably, there are many more installations that will only be needing one network card, called eth0 than not but...

At first I thought that the obvious answer is to make this configurable. Obviously, it is in a "You can edit the files" kind of way but perhaps an install option "Use persistent nic labelling y/n" would be the way to go.

Then I thought some more. The real issue is expecting nic labeling to be more than a convenience. There is zero reason to expect eth0 to mean anything. It's not the persistent coding that's at issue, it's the scripts further down the process that are broken. Assigning 192.168.0.5 to eth0 or setting it DHCP? Wrong! The default action should be the scripts to determine the interface based on rules. Probably by "first available interface" for most set-ups and probably by mac address where more complex set-ups are involved.

That's my take on it anyway. Obviously I think more discussion is warranted. If anyone knows of one ongoing, I'd appreciate a link.

Richard Thomas (7-sd-b) wrote :

Actually, it occurs to me that if the down-process scripts were written how I suggested above, the whole persistent-nic thing would be moot except as a convenience to us silly humans. And that's as it should be.

I guess the meat of the matter boils down to "What to you mean when you say 'eth0'?"

Interface aliases might be another way to do things but I expect that might break a lot of stuff in a lot of nasty ways.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers