virt-customize: dhclient can't get IP address

[Roman Gorshunov]

Ubuntu 18.04 LTS

When running virt-customize against an existing raw image, it seems to hang for a while trying to get IP address, and then gives up. Functionality worked on Ubuntu 17.10, and expected to be working on 18.04. Below are logs from 18.04 and 17.10 of virt-customize running to modify ubuntu-16.04-server-cloudimg-amd64-disk1.img image.

With verbose flag passed, the logs of temporary VM launch:

Ubuntu 18.04 LTS:
+ ip addr add 127.0.0.1/8 brd + dev lo scope host
+ ip link set dev lo up
+ test 1 = 1
++ ls -I all -I default -I lo /proc/sys/net/ipv4/conf
+ iface=eth0
+ touch /etc/fstab
+ dhclient --version
+ dhclient eth0
=== waits for timeout here ===
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
=== NOTE: no ip address above ===
+ ip r
+ cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory

virt-customize works perfectly on Ubuntu 17.10:
+ ip addr add 127.0.0.1/8 brd + dev lo scope host
+ ip link set dev lo up
+ test 1 = 1
++ ls -I all -I default -I lo /proc/sys/net/ipv4/conf
+ iface=eth0
+ touch /etc/fstab
+ dhclient --version
+ dhclient eth0
=== gets IP almost immediately ===
...
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 169.254.2.15/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fec0::5054:ff:fe12:3456/64 scope site tentative mngtmpaddr dynamic
       valid_lft 86400sec preferred_lft 14400sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link
       valid_lft forever preferred_lft forever
=== NOTE: ip address above ===
+ ip r
default via 169.254.2.2 dev eth0
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.2.15
...

Ubuntu 18.04 LTS:
$ dpkg -l | egrep "guestfs|libvirt|qemu"
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii libguestfs-hfsplus:amd64 1:1.36.13-1ubuntu3 amd64 guest disk image management system - HFS+ support
ii libguestfs-perl 1:1.36.13-1ubuntu3 amd64 guest disk image management system - Perl bindings
ii libguestfs-reiserfs:amd64 1:1.36.13-1ubuntu3 amd64 guest disk image management system - ReiserFS support
ii libguestfs-tools 1:1.36.13-1ubuntu3 amd64 guest disk image management system - tools
ii libguestfs-xfs:amd64 1:1.36.13-1ubuntu3 amd64 guest disk image management system - XFS support
ii libguestfs0:amd64 1:1.36.13-1ubuntu3 amd64 guest disk image management system - shared library
ii libsys-virt-perl 4.0.0-1 amd64 Perl module providing an extension for the libvirt library
ii libvirt0:amd64 4.0.0-1ubuntu8 amd64 library for interfacing with different virtualization systems
ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-system-common 1:2.11+dfsg-1ubuntu7 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 1:2.11+dfsg-1ubuntu7 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.11+dfsg-1ubuntu7 amd64 QEMU utilities
$

$ lsb_release -rd
Description: Ubuntu 18.04 LTS
Release: 18.04
$

Ubuntu 17.10:
$ dpkg -l | egrep "guestfs|libvirt|qemu"
ii ipxe-qemu 1.0.0+git-20161027.b991c67+really20150424.a25a16d-1ubuntu2 all PXE boot firmware - ROM images for qemu
ii libguestfs-hfsplus:amd64 1:1.34.6-7ubuntu1 amd64 guest disk image management system - HFS+ support
ii libguestfs-perl 1:1.34.6-7ubuntu1 amd64 guest disk image management system - Perl bindings
ii libguestfs-reiserfs:amd64 1:1.34.6-7ubuntu1 amd64 guest disk image management system - ReiserFS support
ii libguestfs-tools 1:1.34.6-7ubuntu1 amd64 guest disk image management system - tools
ii libguestfs-xfs:amd64 1:1.34.6-7ubuntu1 amd64 guest disk image management system - XFS support
ii libguestfs0:amd64 1:1.34.6-7ubuntu1 amd64 guest disk image management system - shared library
ii libsys-virt-perl 3.5.0-1build1 amd64 Perl module providing an extension for the libvirt library
ii libvirt0:amd64 3.6.0-1ubuntu6.5 amd64 library for interfacing with different virtualization systems
ii qemu-block-extra:amd64 1:2.10+dfsg-0ubuntu3.5 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-system-common 1:2.10+dfsg-0ubuntu3.5 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.10+dfsg-0ubuntu3.5 amd64 QEMU utilities

$ lsb_release -rd
Description: Ubuntu 17.10
Release: 17.10
$

Thank you.

[Richard Jones (rjones-redhat)]

I'm not sure, but can you try a few simple tests:

$ virt-rescue --scratch --network -v -x
$ virt-rescue --scratch -v -x

Do those commands fail in the same way? Does it make a difference if the
--network option is present?

Also it might be worth using the latest version 1.38.

[Roman Gorshunov (romang)]

Hangs on dhclient on Ubuntu 18.04, and works on 17.10:
$ virt-rescue --scratch --network -v -x

Runs perfectly on both 17.10 and 18.04, but there is no network set up:
$ virt-rescue --scratch -v -x

17.10:
...
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
+ ip r
+ lsmod
...

18.04:
...
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
+ ip r
+ cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
+ lsmod
...

Difference in your two commands can be seen in qemu launch options; in 1st command there are parameters '-netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet', and in 2nd there are no such parameters.

Command below shows same behavior as first command you have suggested:
guestfish -a /dev/null --network -v -x run

[Roman Gorshunov (romang)]

Setting 'export LIBGUESTFS_BACKEND=direct' does not help. I will try to find a way to build 1.38 for Ubuntu and re-try.
Thank you, Richard.

[Richard Jones (rjones-redhat)]

The difference with and without --network is in the qemu command line, basically whether
we set up a user network or not:

https://github.com/libguestfs/libguestfs/blob/bd4e99cc134dd20cb1f16ac11e44810de30b973a/lib/launch-direct.c#L674

I'm guessing that ‘dhclient eth0’ only "works" without the network because it fails to find eth0.

Fairly sure this will be some problem with the Ubuntu dhclient program.

One thing you could do (building from libguestfs source) is edit `appliance/init` and `appliance/packagelist.in`. In the first file put `strace` in front of the `dhclient` line. In the second file, add `strace` so the strace package gets added to the appliance.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gonéri Le Bouder (goneri)]

The networking is ok, but the /etc/resolv.conf is deleted before every command execution. As a workaround, you can rewrite it and chain a series of commands:

virt-sysprep -v -x -a CentOS-7-x86_64-GenericCloud.qcow2 --network --run-command 'echo "nameserver 169.254.2.3" > /etc/resolv.conf; yum update -y'