cloud-init dhclient apparmor denied with noexec on /var/tmp

[Tom Kopchak]

Hello - we are seeing an issue on multiple Azure hosts where there is a long delay during bootup. This appears to be related to an apparmor issue with dhclient executed via cloud-init when /var is mounted noexec. Because /var is noexec, the original dhclient is executed rather than the copy in /var/tmp/cloud-init, which causes the AppArmor profile to be applied.

This prevents the instance from being able to record the DHCP lease information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents the instance from being able to obtain goalstate information, and with cloud-init 21.2-3 or later, results in an extended delay during boot due to a recent change in azure.py (https://github.com/canonical/cloud-init/pull/842).

This issue does not occur in default Ubuntu installations (including the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-init behaves differently, copying /usr/sbin/dhclient to /var/tmp/cloud-init/cloud-init-dhcp-xxxxx/dhclient when /var allows executables, and the apparmor profiles then do not apply to the copied executable.

The syslog will show the following entry when the instance boots up:
cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid

The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected:
azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

With the timeouts in azure.py described above, the instance will not boot for around 20 minutes until all 240 connection attempts are completed.

This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process:
type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient
/var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw,
/var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw,

This allows dhclient executed via cloud-init to write the dhclient.pid and dhcp.leases files to /var/tmp/cloud-init and the instance to boot normally.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel Richard G. (skunk)]

This message...

    type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

...is actually for a different issue, discussed at LP: #1918410.