permission denied when executing dhclient in Ec2 datasource

[Scott Moser]

in the ec2 datasource, cloud-init runs dhclient from a tmp file in order to avoid apparmor restrictions and side affects.

In a change for bug 1707222 we started using /run/cloud-init for tmpfiles.
/run is mounted noexec. See example:

$ sudo /run/cloud-init/tmp/dhclient -1 -v -lf /run/cloud-init/tmp/cloud-init-dhcp-bs6g4xkw/dhcp.leases -pf /run/cloud-init/tmp/cloud-init-dhcp-bs6g4xkw/dhclient.pid eth0 -sf /bin/true
sudo: unable to execute /run/cloud-init/tmp/dhclient: Permission denied

So, we need a tmp file in a place that allows execution.

Related bugs:
 * bug 1709772: Enable ipv6 support on EC2
 * bug 1707222: usage of /tmp during boot is not safe may get files deleted.
 * bug 1717627: permission denied executing dhclient from /run

[Dimitri John Ledkov (xnox)]

I would hope that
$ sudo /sbin/dhclient -1 -v -lf /run/cloud-init/tmp/cloud-init-dhcp-bs6g4xkw/dhcp.leases -pf /run/cloud-init/tmp/cloud-init-dhcp-bs6g4xkw/dhclient.pid eth0 -sf /bin/true

Should work fine.

And if /run/cloud-init/tmp/dhclient is not just a copy of /sbin/dhclient, then it should be shipped in the package as /usr/lib/cloud-init/dhclient or some-such. With an apparmor profile applied to it.

dhclient is something that could be remotely exploited, thus protecting it - even if for cloud-init Ec2Local initialisation would be nice.

Can you attach the contents of /run/cloud-init/tmp/dhclient ? Can you elaborate on "in order to avoid apparmor restrictions and side affects"? Is this in fact a bug in Ubuntu stock apparmor profiles?

[Scott Moser (smoser)]

Cloud-init has two stages that probe/discover a datasource (cloud-platform)
and obtain information on that. They are 'local' and 'network'. The local
phase is guaranteed to run before system networking comes up and can read
networking configuration from the datasource.

For further documentation see
 http://cloudinit.readthedocs.io/en/latest/topics/boot.html

Some cloud platforms provide network configuration information on a network
resource. Thus, in order to get this network information, cloud-init has
to configure some networking (ipv4 link local on DigitalOcean, dhcp on
EC2) to get information. When it does this, it tears down the ephemeral
networking that it set up, writes the system configuration and lets the
system bring up the persistent/complete network info.

EC2's network based metadata service (available on http://169.254.169.254)
provides network information. In order to get a response from the meta-data
service, the source address of request must be the address that is returned
in a dhcpv4 request. The requirement is then to to do a dhcp request on
a given network device.

Rather than writing our own dhcp client, we chose to do this by utilizing
dhclient and executing it in a way that it did not invoke hooks or otherwise
have side affects on the system. In order to do that, we invoke like
   dhclient -1 -v -lf /tmp/my.leases -sf /bin/true
That is to control where the lease_file is written and to not execute any
scripts.

Our invocation of dhclient causes issues with the default Ubuntu shipped
dhclient apparmor profile as it does not allow execution of /bin/true.
The solution we employed was to skip app armor constraint by copying
/sbin/dhclient to a temporary file and execute it there. Other options
were to
 * disable apparmor temporarily (which would leave scary warning messages
   in the log).
 * ship a wrapper around dhclient and a apparmor profile.

Our most recent issue with the solution we used is bug 1717627. That
was a result of using /run/cloud-init which is mounted noexec rather than
/tmp/ (to fix bug 1707222).

Lastly, the change from using dhclient to using systemd-networkd's dhcp client
will cause a fair amount of fallout across Ubuntu. That will cause problems
with cloud-init on Azure. Many things either relied on dhclient hooks
or read its leases file. I've opened bug 1717983 to track that.

Summary of generally involved bugs:
 * bug 1709772: Enable ipv6 support on EC2
 * bug 1707222: usage of /tmp during boot is not safe may get files deleted.
 * bug 1717627: permission denied executing dhclient from /run
 * bug 1717983: cloud-init and other programs integrate with dhclient.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-283-g7eb3460b-0ubuntu1

---------------
cloud-init (0.7.9-283-g7eb3460b-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - ec2: Fix maybe_perform_dhcp_discovery to use /var/tmp as a tmpdir
      [Chad Smith] (LP: #1717627)
    - Azure: wait longer for SSH pub keys to arrive.
      [Paul Meyer] (LP: #1717611)

 -- Scott Moser <email address hidden> Mon, 18 Sep 2017 20:41:53 -0400

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3002