initial dhcp has default hostname of ubuntu

[Simon Davy]

When using the normal lxd-bridge, using a dnsmasq instance for dhcp, the initial dhcp is always the hostname 'ubuntu', and this is recorded in the dnsmasq's dhcp leases file.

Presumably the dhcp is done before the container's hostname is set. A restart or dhcp renew seems to put the correct container name in the leases file.

This is a problem when using the dnsmasq for local dns resolving for *.lxd, which is the standard way of doing host dns for containers, as new containers are not dns addressable without a restart or renew.

Is there anyway get the correct hostname in the initial dhcp? Or maybe renew after setting the hostname?

Related Bugs:
 * LXC/LXD Issue https://github.com/lxc/lxd/issues/2244

[Stéphane Graber (stgraber)]

LXD doesn't set the hostname (well, it sets the kernel utsname but that's overridden by the hostname).

Instead LXD just feeds metadata to cloud-init which is then responsible for setting the hostname.
The problem is that cloud-init runs after network was brought up, so the hostname is incorrect at the time of the initial dhcp request.

A container reboot solves it, so would cloud-init setting the hostname before network is brought up or so would systemd re-starting the dhcp client after changing the hostname.

It could be that some of this problem is somehow resolved with recent cloud-init and the network config code but I'm sure Scott can tell us for sure :)

[Simon Davy (bloodearnest)]

Some more info:

lxd is writing the the hostname into cloud init metadata file before boot, so the data is on disk before networks starts up. Whether cloud-init can access this early I am unsure.

Also, I double checked this on yakkety, and the problem remains.

Thanks

[Scott Moser (smoser)]

There are at least 3 ways to fix this:
a.) lxd templates can write /etc/hostname as they write /var/lib/cloud/data/seed/nocloud/meta-data
    then, when the system does a dhcp on its first boot, it will have the /etc/hostname that lxd thinks it should

b.) cloud-init can read the hostname from meta-data and set it before the dhcp.
   This is only really possible recently in cloud-init, but it is now possible.

c.) lxd can control dnsmasq better

'a' and 'b' have the requirement that they container continue to behave, and if a container changes its /etc/hostname and dhcps again, it will then possibly have a new hostname. Basically, that puts the container in charge of declaring its hostname that will appear in dns. That brittle . The failure you see here is fallout of that system being brittle.

if LXD says a system's name is 'cloudy-weather' and something in the container writes 'sunny-day' to /etc/hostname, a subsequent reboot would apparently change the dns entry for that system. Essentially, you're allowing a api by which a container can change its desired hostname by metadata in a dhcp request. That seems an odd api at best.

It also means that if another system was already named 'sunny-day', then you have a collision and a misbehaving system can actually do some harm.

In my head, ultimately the only fix is 'c'.

either 'a' or 'b' will improve the situation, but not fix it.

[Stéphane Graber (stgraber)]

Some comments:

a) Certainly doable, we do it for the non-cloud-init images. Fairly straightforward for CPC to do. We don't do it today because cloud-init does this everywhere else so it felt natural to let cloud-init do it. One potential problem with doing it through the template is that the user may want to set a hostname that's different from the container name. So we'd first set the container name as the hostname and cloud-init would then change it again to the name the user provided through user-data.

If done this way, we'd mark the template as triggering on "create" and "copy" so it'd only get templated on initial container creation or when cloning the container. Those are the same conditions under which we write the cloud-init seed data.

b) I think that's what folks expect cloud-init to do.

c) We will eventually do this, possibly this cycle still. However note that this isn't a perfect solution either as again, the container name and expected hostname may differ. This is most visible with nova-lxd where the containers are all named instance-XYZ but the hostname in the container is to be set to whatever the user wants.

[Scott Moser (smoser)]

In 'c', i think dnsmasq probably is not involved. Instead, there is a system in control of the dns and it can chose to listen to the hostname requested over dhcp or not. openstack probably has better interfaces for setting dns entries than having a client do a dhcp.

[Scott Moser (smoser)]

We can do 'b', and I agree that done well its probably the right thing to do generally speaking. But without 'c' stuff is generally quite brittle.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Claudio Gardini (cgardini)]

That also affects deploys of Kubernetes on Canonical distribution with Juju on a local vSphere cluster.
Creating new kubernetes workers (via "juju add-unit kubernetes-worker") all the hosts are recorded on DHCP server with name "ububtu".
If a reboot or manual DHCP lease renew is not operated manually, the nodes are not resolvable by name and there also side issues with Kubernetes .

[Anastasia (anastasia-macmood)]

Potentially, a solution here might also address the bug # 1746358.

[Anastasia (anastasia-macmood)]

From Scott's comments and John's comments in the linked bug, this does not seem to be a Juju but cloud-init issue only.

[Chris Gregan (cgregan)]

Impacting on site customer deployment....field high added

[David Britton (dpb)]

Upstream Issue: https://github.com/lxc/lxd/issues/2244

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2691