Ubuntu eks AMI's resolv.conf puts 032 instead of space when multiple search domains are configured in amazon dhcp options

[Suren Nihalani]

VPC's dhcp options set to multiple domains and then start a cluster with EKS AMI.

cat /etc/resolv.conf

```
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.0.50.4
nameserver 10.0.50.5
nameserver 10.0.50.6
search domain1.net032ec2.internal032domain2.net
```

[Cody Shepherd (codyshepherd)]

Reproduced with 18.04 standard AMI; confirmed bug not present in 16.04.
This file is written out by systemd-resolved, which was introduced between 16.04 and 18.04, which explains the change in behavior.

Here's the same issue being reported on CoreOS as well:
https://github.com/coreos/bugs/issues/1934

We are doing more research to see if there's a path forward to support this behavior, though there is some disagreement on what the correct behavior is here based on the DHCP spec.

We will continue to investigate and provide updates on this bug.

[Suren Nihalani (snihalani4)]

Cool. Thank you @codeshepherd.

For other people who come across this, in the meantime (while the bug stays open), I ended up writing my own dhclient hook to manage resolv.conf myself

[Cody Shepherd (codyshepherd)]

As an update, we've determined this is a bug with Amazon's DHCP implementation. We have filed a bug with them and will update this bug once the issue is resolved.

[Suren Nihalani (snihalani4)]

Is there a public link where this is being discussed?

[GGrandes (ggrandes)]

My workaround solution (could be better):

# Enable dhclient-script
FILE="/etc/dhcp/dhclient.conf"
dpkg-divert --package "isc-dhcp-client" \
  --add "$FILE"
sed -i -e 's|#script.*|script "/sbin/dhclient-script";|g' $FILE

# Disabled resolved hook
FILE="/etc/dhcp/dhclient-enter-hooks.d/resolved"
dpkg-divert --package "isc-dhcp-client" \
  --add "$FILE"
mv $FILE $FILE.distrib
touch $FILE
chattr +i $FILE

# Rewrite parameters
cat >> /etc/dhcp/dhclient-enter-hooks.d/aaaa <<"END"
#!/bin/sh
export new_domain_search="$new_domain_name $new_domain_search"
export new_domain_name=
END
cat >> /etc/dhcp/dhclient-exit-hooks.d/aaaa <<"END"
#!/bin/sh
export new_domain_search="$new_domain_name $new_domain_search"
export new_domain_name=
END

[Clement Guillaume (guillaumecle)]

Is the AWS bug resolved? or is there any other update?

[Glenn Prinsen (glenn-w)]

This is not an AWS bug specifically, and occurs with a 20.04 Server install (but not Desktop 20.04 install).

Whenever an interface receives multiple search domains via DHCP, they will have "032" (the 3 numerical characters) instead of a space character.

systemd package version is 245.4-4ubuntu3.11

[Robert C Jennings (rcj)]

@glenn-w, are you seeing this with a DHCP server sending multiple domains in DHCP option 15 (which is not RFC2132 compliant) or are you reporting a new bug where systemd-resolved is broken when the DHCP server provides multiple hosts via DHCP option 119? If this is an issue with DHCP option 119 then a new bug should be filed[0] against systemd with steps to recreate instead of reusing this bug which is specific to AWS EKS images and DHCP option 15 non-compliant server behavior.

While this issue is not specific to AWS, AWS's DNS server does exhibit this behavior as acknowledged in their documentation[1]. The underlying cause of the issue in this bug report is a DHCP server not complying with the RFC2132 and sending multiple domains in option 15 rather than using DHCP option 119 for multiple domains. An excellent write-up is available on Igo Cicimov's blog[2].

[0] https://bugs.launchpad.net/ubuntu/+source/systemd/+filebug
[1] https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html#DHCPOptionSets
[2] https://icicimov.github.io/blog/devops/aws-ubuntu-systemd-resolved-dns-issues/

[Glenn Prinsen (glenn-w)]

@rcj Thank you for pointing it out. You're correct this was an error in my DHCP server configuration which I've now corrected and it's working correctly. My apologies and thanks for the pointers!

[Tired Sysadmin (wearyofallthiscrap)]

For people finding this page via Google, or via other bug databases linking here, the AWS documentation from comment 8 footnote 1 is now at

https://docs.aws.amazon.com/vpc/latest/userguide/DHCPOptionSet.html#CreatingaDHCPOptionSet

instead, as AWS shuffled around their URLs a while after (ironically not providing anything other than the "latest" element in the URL to create a stable reference... grumble grumble).