[Azure] User's customized dns search domains is over-written as re-config network on every boot

[Huijuan Zhao]

Description of problem:
This issue only exists on Azure cloud-init.
With the 22.1 version, we add dns search domains via nmcli con modify 'System eth0' +ipv4.dns-search domain.tld, after reboot, the config is over-written by cloud-init(the changes to /etc/sysconfig/network-scripts/ifcfg-eth0 get deleted, they used to be persistent), as Azure apply networking config on every BOOT[1] since cloud-init-22.1

[1] https://github.com/canonical/cloud-init/pull/1023

But some users prefer to persist the customized network config(e.g. DNS) after reboot, so could you please help to check how to enhance this patch[1] to meet customer's requirement[2]?

[2] User's requirements:
We add additional search domains to certain servers, mostly because they communicate with legacy urls in our company without using fqdn.

Additional info:
Below[3] is a workaround to persist user's customized DNS. But as all the customized network configs will be over-written after reboot, so maybe we need to consider how to avoid it.

[3] Adding the customized DNS to /etc/dhcp/dhclient.conf,
e.g.
Modify /etc/dhcp/dhclient.conf
timeout 300;
append domain-search "searchdomain1.com";
append domain-search "searchdomain2.com";
append domain-name-servers 1.1.1.1;

Reboot the VM and then check the dns information, the modifications are not overwritten.
# cat /etc/resolv.conf
# Generated by NetworkManager
search searchdomain1.com searchdomain2.com
nameserver 1.1.1.1

Test Veision: 22.1

How reproducible:
Modify dns search domains, reboot

Steps to Reproduce:
1. nmcli con modify 'System eth0' +ipv4.dns-search example.com
2. systemctl reboot

Actual results:
$ grep example.com /etc/resolv.conf
[No result]

Expected results:
$ grep example.com /etc/resolv.conf
search example.com

[James Falcon (falcojr)]

Re-applying networking every boot is the expected behavior. It worked this way in the past for Ubuntu but was inconsistently applied across distros. It is now consistent across distros, but it does represent a change in behavior for those that weren't on Ubuntu distros. Because of how Azure wants their instances initialized, this behavior can't be reverted upstream. However, there shouldn't have been an expectation of changes to these files persisting. At the top of /etc/sysconfig/network-scripts/ifcfg-eth0, it even says
"# Created by cloud-init on instance boot automatically, do not edit."

Without knowing your distro/networking tools, I can't know the best way to workaround the issue, but most tools should provide a way of manually specifying a search domain. For example, when using systemd-resolved, you should be able to add search domains to /etc/systemd/resolved.conf or /etc/systemd/resolved.conf.d/ . Beyond that, is there a problem with your [3] option you mentioned?

[Huijuan Zhao (huijuanzhao)]

Thank you James for the quick reply.

Understand the patch intent, is it possible for us to enhance the patch to persist manually configuration? e.g. Put the manually config to the user-data? As you know, customers got used to the config, it is a regression for them to change the config method.

The distro/networking tools are RHEL/NetworkManager. And there are indeed some workarounds to persist the specific DNS manually config, option[3] is also works, but is this recommended workaround for customers in this scenario? And except DNS, all the manually configs(if have) will be over-written after reboot. So for the workaround, do you have any suggestions?

I just think about the file in /etc/cloud/cloud.cfd.d/network_xx.cfg, we can ask users disable network in the file, but that maybe cause other issues, e.g. can not set new network configs automatic via cloud-init.

How about adding users' manually network config to the file /etc/cloud/cloud.cfd.d/network_xx.cfg?

I just would like to find a cloud-init recommended and long-term solution/workaround for the scenario/requirement.

[James Falcon (falcojr)]

> I just think about the file in /etc/cloud/cloud.cfd.d/network_xx.cfg, we can ask users disable network in the file, but that maybe cause other issues, e.g. can not set new network configs automatic via cloud-init.
How about adding users' manually network config to the file /etc/cloud/cloud.cfd.d/network_xx.cfg?

Both of these options probably won't work in a cloud environment as the cloud is providing network information that cloud-init needs to know to setup networking correctly. It could be possible to do that after first boot, but as you say, any networking changes from the cloud would likely leave the instance in an unreachable state.

> And there are indeed some workarounds to persist the specific DNS manually config, option[3] is also works, but is this recommended workaround for customers in this scenario?

It depends on the tooling available to your machine, but yes, this is one of the persistent ways to make such a change, at least until dhclient is no longer supported. The other way is to use the systemd-resolved configuration file if it being used in your configuration.

Do either of these cloud-configs work for you?
Using systemd-resolved:
#cloud-config
write_files:
  - path: /etc/systemd/resolved.conf.d/dns_servers.conf
    content: |
      [Resolve]
      DNS=1.1.1.1
      Domains=example.com
    owner: root:root
    permissions: '0o644'
    encoding: text/plain
    append: false
    defer: false

Using dhclient:
#cloud-config
write_files:
  - path: /etc/dhcp/dhclient.conf
    content: |
      append domain-search "example.com";
      append domain-name-servers 1.1.1.1;
    owner: root:root
    encoding: text/plain
    append: true
    defer: false

[Huijuan Zhao (huijuanzhao)]

> Do either of these cloud-configs work for you?
Using dhclient:
$ cat /etc/cloud/cloud.cfg.d/network_cu.cfg
#cloud-config
write_files:
  - path: /etc/dhcp/dhclient.conf
    content: |
      append domain-search "example.com";
      append domain-name-servers 1.1.1.1;
    owner: root:root
    encoding: text/plain
    append: true
    defer: false

We use /etc/dhcp/dhclient.conf in my distro(rhel). I tested this scenario, it only works during instance first boot. It means users need to add this file /etc/cloud/cloud.cfg.d/network_cu.cfg to image, deploy and boot instance(write content to dhclient.conf), then reboot instance, now can make the new added DNS available(adding to /etc/resolv.conf).

But in this user's scenario, they would like to modify DNS in a booted instance, then reboot instance, make the modified DNS available.

So seems using /etc/cloud/cloud.cfg.d/xx.cfg is NOT a good workaround for this scenario.

Any thoughts?

Thanks!

[James Falcon (falcojr)]

"But in this user's scenario, they would like to modify DNS in a booted instance, then reboot instance, make the modified DNS available."

I see. I was under the impression they wanted to do this first boot. If they want to make changes after first boot has happened, then I think the current workaround (the changes to /etc/dhcp/dhclient.conf) makes the most sense.

[Huijuan Zhao (huijuanzhao)]

Thank you James.

I will doc the workarounds for users if we can not enhance the patch to meet the specific requirements.

Thanks!
Huijuan

[James Falcon (falcojr)]

Sounds good. Since we have a resolution, I'm going to go ahead and close this bug as "won't fix", but feel free to continue the conversation if you have any more concerns.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4065