execve (/bin/true, ...): Permission denied printed,when cloud-init-local run

[Dong Liang]

In cloud-init-local stage , using dhclient command to discovery ip and getting metadata from meta-server.
The command which cloud-init-local execute is '/usr/sbin/dhclient -1 -v -lf /run/dhclient.lease -pf /run/dhclient.pid eth0 -sf /bin/true'. However, when the apparmor service is loaded, the kernel prevents the /usr/sbin/dhclient command from executing scripts other than /sbin/dhclient-script. And when the -sf parameter is used in cloud-init-local to specify the execution of /bin/true, this error occurs during the execve call in the dhclient process.

lsb_release -rd:
Description: Ubuntu 22.04.2 LTS
Release: 22.04

cloud-init : 23.1.2-0ubuntu0~22.04.1

[Calvin Mwadime Makokha (calvinmwadime)]

Thanks for raising the issue.
Could you please provide us with the cloud-init logs using:

sudo cloud-init collect-logs

and upload the resulting file.

The logs will help us in understanding what is leading to this issue.
Furthermore, Please help us with any additional configurations you have for apparmor on your system.
I am marking this as 'incomplete', change to 'new' when you update the bug with additional content requested.

[Dong Liang (qidong-ld)]

apparmor version：3.0.4-2ubuntu2.2
the sbin.dhclient config file which isc-dhcp-client deb package provided is as follows：

dpkg -S /etc/apparmor.d/sbin.dhclient
isc-dhcp-client: /etc/apparmor.d/sbin.dhclient

 cat sbin.dhclient
# vim:syntax=apparmor
#include <tunables/global>

/{,usr/}sbin/dhclient flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/openssl>

  capability net_bind_service,
  capability net_raw,
  capability dac_override,
  capability net_admin,

  network packet,
  network raw,

  @{PROC}/[0-9]*/net/ r,
  @{PROC}/[0-9]*/net/** r,

  # dhclient wants to update its threads with functional names
  # https://gitlab.com/apparmor/apparmor/-/merge_requests/730
  # see LP: #1918410
  owner @{PROC}/@{pid}/task/[0-9]*/comm rw,

  # LP: #1926139
  @{PROC}/cmdline r,

  /{,usr/}sbin/dhclient mr,
  # LP: #1197484 and LP: #1202203 - why is this needed? :(
  /{,usr/}bin/bash mr,

  /etc/dhclient.conf r,
  /etc/dhcp/ r,
  /etc/dhcp/** r,

  /var/lib/dhcp{,3}/dhclient* lrw,
  /{,var/}run/dhclient*.pid lrw,
  /{,var/}run/dhclient*.lease* lrw,

  # NetworkManager
  /{,var/}run/nm*conf r,
  /{,var/}run/sendsigs.omit.d/network-manager.dhclient*.pid lrw,
  /{,var/}run/NetworkManager/dhclient*.pid lrw,
  /var/lib/NetworkManager/dhclient*.conf lrw,
  /var/lib/NetworkManager/dhclient*.lease* lrw,
  signal (receive) peer=/usr/sbin/NetworkManager,
  ptrace (readby) peer=/usr/sbin/NetworkManager,

  # connman
  /{,var/}run/connman/dhclient*.pid lrw,
  /{,var/}run/connman/dhclient*.leases lrw,

  # synce-hal
  /usr/share/synce-hal/dhclient.conf r,

  # if there is a custom script, let it run unconfined
  /etc/dhcp/dhclient-script Uxr,

  # The dhclient-script shell script sources other shell scripts rather than
  # executing them, so we can't just use a separate profile for dhclient-script
  # with 'Uxr' on the hook scripts. However, for the long-running dhclient3
  # daemon to run arbitrary code via /sbin/dhclient-script, it would need to be
  # able to subvert dhclient-script or write to the hooks.d directories. As
  # such, if the dhclient3 daemon is subverted, this effectively limits it to
  # only being able to run the hooks scripts.
  /{,usr/}sbin/dhclient-script Uxr,

  # Run the ELF executables under their own unrestricted profiles
  /usr/lib/NetworkManager/nm-dhcp-client.action Pxrm,
  /usr/lib/connman/scripts/dhclient-script Pxrm,

  # Support the new executable helper from NetworkManager.
  /usr/lib/NetworkManager/nm-dhcp-helper Pxrm,
  signal (receive) peer=/usr/lib/NetworkManager/nm-dhcp-helper,

  # Site-specific additions and overrides. See local/README for details.
  #include <local/sbin.dhclient>
}

/usr/lib/NetworkManager/nm-dhcp-client.action {
  #include <abstractions/base>
  #include <abstractions/dbus>
  /usr/lib/NetworkManager/nm-dhcp-client.action mr,

  /var/lib/NetworkManager/*lease r,
  signal (receive) peer=/usr/sbin/NetworkManager,
  ptrace (readby) peer=/usr/sbin/NetworkManager,
  network inet dgram,
  network inet6 dgram,
}

/usr/lib/NetworkManager/nm-dhcp-helper {
  #include <abstractions/base>
  #include ...

[Christian Ehrhardt  (paelzer)]

Thank,
Calvin and I looked at it and you are right.

This is in cloud-init since [1] and used -sf as in [2] to avoid any side effects by the content of that file.
=> "... If unspecified, the default /sbin/dhclient-script is used. ..."

I think it should be safe to allow execution of that as a common pattern to avoid falling back to the default.

For the severity we need to find out, if
a) call dhclient with -sf /bin/true to do nothing
isn't just the same as
b) call dhclient with -sf /bin/true, get denied, but do everything else the same

If they are the same then this would only cause the annoying log message.
If they actually behave different

We can recreate this in a VM:

ubuntu@l:~$ sudo dhclient -v -1 -sf /bin/true
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

execve (/bin/true, ...): Permission denied
Listening on LPF/enp1s0/52:54:00:be:7a:d2
Sending on LPF/enp1s0/52:54:00:be:7a:d2
Sending on Socket/fallback
xid: warning: no netdev with useable HWADDR found for seed's uniqueness enforcement
xid: rand init seed (0x649ac969) built using gethostid
DHCPREQUEST for 192.168.122.239 on enp1s0 to 255.255.255.255 port 67 (xid=0x7ee9d4f5)
DHCPACK of 192.168.122.239 from 192.168.122.1 (xid=0xf5d4e97e)
execve (/bin/true, ...): Permission denied
bound to 192.168.122.239 -- renewal in 1781 seconds.
ubuntu@l:~$ echo $?
0

We see it is denied, but otherwise happy.

So my point is.
We try to use -sf /bin/true so it does not execute anything with a side-effect
Being denied by apparmor means no side-effect happens.
So - as assumed - the only "impact" is the annoying log message.

[1]: https://git.launchpad.net/cloud-init/commit/?id=d5f855dd96ccbea77f61b0515b574ad2c43d116d
[2]: https://manpages.ubuntu.com/manpages/jammy/man8/dhclient.8.html

[Christian Ehrhardt  (paelzer)]

Looking at most recent isc-dhcp

I see:

 65 # Since dhclient doesn't provide the option to disable hooks, which is
 66 # desireable in some cases, executing /bin/true as the script file suffices
 67 /{,usr/}bin/true ixr,

Which should actually allow it, right?

But the apparmor profile in comment #3 has not.
:-)

Checking different releases.
- fixed in Mantic
- not fixed in Lunar
- not fixed in Debian

Change came in via no other than:

 115 isc-dhcp (4.4.3-P1-1ubuntu2) mantic; urgency=medium
 116
 117 * debian/apparmor/sbin.dhclient: Allow disabling dhclient hooks. LP: #2011628
 118
 119 -- Brett Holman <email address hidden> Fri, 17 Mar 2023 15:38:35 -0600

So we are aware of this.
Therefore this bug here is actually a duplicate of bug 2011628