isc-dhcp dhclient listens on extra random ports

[Rodney Beede]

[Impact]

In trusty, there is only 1 version of dhclient, including #define NSUPDATE, which introduce DDNS functionnality.
The DDNS functionnality, generate 2 random extra ports between 1024-65535.

Impact reported by users :

"One impact of these random ports is that security hardening becomes more difficult. The purpose of these random ports and security implications are unknown."
"We have software that was using one of the lower udp ports but it happened to collide with dhclient which seems to allocate 2 random ports."

There is a randomization mechanism in libdns that prevent dhclient to take the sysctl values into account (net.ipv4.ip_local_port_range & net.ipv4.ip_local_reserved_ports) to workaround this, and after discussion isc-dhcp upstream doesn't want to rely on kernel for randomization.

There is no realtime configuration to disable the feature or workaround this. The only possible way is at compile time.

I also talk with upstream maintainers, and there is no way they will accept to reduce the range (1024-65535) for security reason. Reducing the port range may facilitate the spoofing.

Xenial has separated dhclient in two packages :

isc-dhcp-client pkg : dhclient with DDNS functionality disabled (no random extra ports)
isc-dhcp-client-ddns : dhclient with DDNS functionality enabled (with random extra ports)

The goal here is to reproduce the same situation in Trusty, for this bug to be less painful for at least users that doesn't require DDNS functionnality.

[Test Case]

Run a Trusty image with following package :
isc-dhcp-client
isc-dhcp-common

```
dhclient 1110 root 6u IPv4 11535 0t0 UDP *:bootpc
dhclient 1110 root 20u IPv4 11516 0t0 UDP *:64589 # <----------- extra random port
dhclient 1110 root 21u IPv6 11517 0t0 UDP *:7749 # <----------- extra random port
```

[Regression Potential]

I did the split such that Trusty users will automatically get "isc-dhcp-client-ddns" installed but users bothered by this bug will have the option to switch to "isc-dhcp-client-noddns".

Existing Trusty users can continue to use this DDNS functionality after the SRU without any necessary intervention.

With isc-dhcp-client:
dhclient 1110 root 6u IPv4 11535 0t0 UDP *:bootpc
dhclient 1110 root 20u IPv4 11516 0t0 UDP *:64589 # <----------- extra random port
dhclient 1110 root 21u IPv6 11517 0t0 UDP *:7749 # <----------- extra random port

With isc-dhcp-client-noddns :
dhclient 1110 root 6u IPv4 11535 0t0 UDP *:bootpc

Xenial also has both distinct dhclient binary package but in the opposite way. We have decided to use the opposite way approach for not impacting actual Trusty users by changing the nature of isc-dhcp-client itself.

Caribou and I, slashd, have also tested a couple of release upgrades from Trusty to Xenial with both scenarios :
1 - Trusty upgrade to Xenial with "isc-dhcp-client-ddns"
2 - Trusty upgrade to Xenial with "isc-dhcp-client-noddns"

and both scenarios worked as expected for caribou and I. (See comment #42)

Results :
===
** Upgrade tested with isc-dhcp-client **

# dpkg -l
ii isc-dhcp-client 4.2.4-7ubuntu12.8 amd64 ISC DHCP client
ii isc-dhcp-common 4.2.4-7ubuntu12.8 amd64 common files used by all the isc-dhcp* packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:20114 0.0.0.0:* 632/dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 632/dhclient
udp6 0 0 :::52249 :::* 632/dhclient

After successful upgrade Trusty (14.04.5) -> Xenial (16.04.2)
ii isc-dhcp-client 4.3.3-5ubuntu12.7 amd64 DHCP client for automatically obtaining an IP address
ii isc-dhcp-common 4.3.3-5ubuntu12.7 amd64 common files used by all of the isc-dhcp packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 633/dhclient

** Upgrade tested with isc-dhcp-noddns (4.2.4-7ubuntu12.9) **

# dpkg -l
ii isc-dhcp-client-noddns 4.2.4-7ubuntu12.9 amd64 Dynamic DNS (DDNS) disabled DHCP client

# netstat -anputa | grep dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 682/dhclient

After successful upgrade Trusty (14.04.5) -> Xenial (16.04.2)

# dpkg -l | grep -i dhcp
ii isc-dhcp-client 4.3.3-5ubuntu12.7 amd64 DHCP client for automatically obtaining an IP address
rc isc-dhcp-client-noddns 4.2.4-7ubuntu12.9 amd64 Dynamic DNS (DDNS) disabled DHCP client
ii isc-dhcp-common 4.3.3-5ubuntu12.6 amd64 common files used by all of the isc-dhcp packages

# netstat -anputa | grep dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 639/dhclient
===

For ~ubuntu-sru, the above tests/scenarios will also be part of the testing phase once the package will be found in -proposed and I'll make sure they all succeed before tagging the bug to "verfication-done".

[Other Info]

 * See : https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1176046/comments/19 to look at my discussion with rbasak on if that approach would be acceptable for SRU.

[Original Description]

Ubuntu 13.04 Server 64-bit. Fresh install. Only one network adapter.

dhclient process is listening on two randomly chosen udp ports in addition to the usual port 68. This appears to be a bug in the discovery code for probing information on interfaces in the system.

Initial research of the code also suggested omapi, but adding omapi port 9999 to /etc/dhcp/dhclient.conf only opened a forth port with the two random udp ports still enabled.

Version of included distro dhclient was 4.2.4. I also tested with the latest isc-dhclient-4.2.5-P1 and got the same results.

Debian has the same bug: http://forums.debian.net/viewtopic.php?f=10&t=95273&p=495605#p495605

One impact of these random ports is that security hardening becomes more difficult. The purpose of these random ports and security implications are unknown.

Example netstat -lnp output:

udp 0 0 0.0.0.0:21117 0.0.0.0:* 2659/dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 2659/dhclient
udp6 0 0 :::45664 :::* 2659/dhclient

[Rodney Beede (business2008+launchpad)]

I found the cause from a helpful thread at http://forums.debian.net/viewtopic.php?f=10&t=95273

The NSUPDATE dns functionality in dhclient automatically causes it to listen on two random UDP ports. This could pose a security issue since the client will accept packets on these ports.

The current fix is to modify the source code to disable the functionality as per the thread mentioned above. Doing so stops it from listening on the random ports.

I have filed a bug with ISC, bug number [ISC-Bugs #33377] asking for documentation about this feature (none exists currently about why the ports are opened) as well as a run-time configuration option with a default value to disable it.

[Yves-Alexis Perez (corsac)]

Can you provide a link to the ISC issue?

[Stéphane Graber (stgraber)]

The ISC bug tracker is unfortunately private, so you can file bugs but you can't get updates about them or refer to them publicly.

That's a source of major frustration as all distros end up filling the same bugs over and over again without being able to collaborate and add information to existing reports...

[Michael Bazzinotti (mbazzinotti)]

I am experiencing a similar or same issue on Ubuntu 14.04 Server 64-bit.

The Dhclient is listening on both ports 68 and 42370. I have no experience with dhclient so I cannot decide if this is normal behaviour. Why the extra port?

[Tim Utschig (tim-tetro)]

Apparently this is still happening in Ubuntu 15.04 "vivid".

Any updates?

[brijan (brijan30309)]

I am still facing same issue in centos 15.10

[glinsvad (glinsvad)]

I can't reproduce this on Ubuntu 16.04 "xenial" (dhclient only listens on port 68) but it remains an issue on Ubuntu 15.10 "wily".

[David Coronel (davecore)]

I confirm I have the same behavior in Ubuntu 14.04.5 LTS with isc-dhcp-client version 4.2.4-7ubuntu12.7

I recompiled the isc-dhcp-client with the fix from http://forums.debian.net/viewtopic.php?f=10&t=95273 and I no longer have the issue:

BEFORE:

root@trustydhclient:~# lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 521 root 5u IPv4 7553 0t0 UDP *:bootpc
dhclient 521 root 20u IPv4 7479 0t0 UDP *:50522
dhclient 521 root 21u IPv6 7480 0t0 UDP *:17754

AFTER:

root@trustydhclient:~# lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 524 root 5u IPv4 7613 0t0 UDP *:bootpc

[Eric Desrochers (slashd)]

Sound similar to debian bug : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=712503

The situation doesn't occur on Xenial (16.04 LTS), but present in Trusty (14.04) as a fix has been first introduce in Ubuntu with version : isc-dhcp (4.3.3-2)

Trusty :
#netstat -tulp | grep -i dhclient
udp 0 0 *:bootpc *:* 23402/dhclient
udp 0 0 *:bootpc *:* 1589/dhclient
udp 0 0 *:33749 *:* 1589/dhclient
udp 0 0 *:11662 *:* 23402/dhclient
udp6 0 0 [::]:7082 [::]:* 1589/dhclient
udp6 0 0 [::]:12668 [::]:* 23402/dhclient

Xenial:
# netstat -tulp | grep -i dhclient
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 *:bootpc *:* 142/dhclient

# changelog
isc-dhcp (4.3.3-2) unstable; urgency=medium
...
 ===> * Disable NSUPDATE (closes: #712503). <==
...

[Eric Desrochers (slashd)]

Then NSUPDATE has been re-enable with debian bug :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798237

# changelog
isc-dhcp (4.3.3-4) unstable; urgency=medium

  * Reenable NSUPDATE (closes: #798237)

I will look at both changes, but most likely the one in #798237 seems more suitable for a wider audience, since the change in #712503 cause some unwanted situation for certain debian users.

Eric

[Eric Desrochers (slashd)]

NSUPDATE is disabled in Xenial, will keep the same approach for Trusty and then test.

More details can be found here :

xenial/isc-dhcp-4.3.3/debian/patches/disable-nsupdate.patch
description: undefine NSUPDATE
author: cchip
origin: http://forums.debian.net/viewtopic.php?f=10&t=95273
bug-debian: https://bugs.debian.org/712503

--- a/includes/site.h
+++ b/includes/site.h
@@ -121,7 +121,7 @@

 /* Define this if you want DNS update functionality to be available. */

-#define NSUPDATE
+/* #define NSUPDATE */

 /* Define this if you want to enable the DHCP server attempting to
    find a nameserver to use for DDNS updates. */

[Eric Desrochers (slashd)]

Applying the Xenial "disable-nsupdate.patch" result in a failling build in Trusty.

/<<PKGBUILDDIR>>/server/dhcpd.c:498: undefined reference to `trace_ddns_init'
dhcpd-failover.o: In function `dhcp_failover_process_bind_update':
/<<PKGBUILDDIR>>/server/failover.c:5225: undefined reference to `ddns_removals'

Seems like failover rely on this to work properly.

[Eric Desrochers (slashd)]

The following debian[1] forum indicate that failover and tracing need to be disabled in order to disable NSUPDATE as well and I confirmed the steps, otherwise the build fails.

I'm not too sure if this is the right way to fix this problem as I don't want to introduce new issues that will impact other users.

Additionally, Debian had to re-enable NSUPDATE after disabling it due to some issues

* Reenable NSUPDATE (closes: #798237)

Will need to look at it more carefully, and talk with the foundation team maybe.

[1] - http://forums.debian.net/viewtopic.php?f=10&t=95273

Eric

[Eric Desrochers (slashd)]

Rectification, after more investigation here what is happening :

Trusty got the random port right away because NSUPDATE is enable at compile time in package "isc-dhcp-client"

In Xenial and late (including debian) with commit [b1d5bab630a29f2ca94c458f9402aa43cc5975c4], a new pkg binary package is introduced "isc-dhcp-client-ddns" set as optional.

#debian/control:
+Package: isc-dhcp-client-ddns
+Priority: optional
+Architecture: any
+Depends:
+ ${misc:Depends},
+ ${shlibs:Depends},
+ isc-dhcp-client (= ${binary:Version}),
+Enhances: isc-dhcp-client
+Description: Dynamic DNS (DDNS) enabled DHCP client
+ This a Dynamic DNS enabled version of the DHCP client.

So basically the random port still exist in Xenial and late, but it requires, the "isc-dhcp-client-ddns" package (optional) to be installed. So basically, it is split probably to accommodate the folks having this problem, and in the same time it allows peoples that really needs this feature to use it by installing this additionnal new package.

On Xenial :
** Without "isc-dhcp-client-ddns" **
# lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 140 root 6u IPv4 344477 0t0 UDP *:bootpc

** With "isc-dhcp-client-ddns" **
# lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 140 root 6u IPv4 336839 0t0 UDP *:bootpc
dhclient 140 root 20u IPv4 336827 0t0 UDP *:65239
dhclient 140 root 21u IPv6 336828 0t0 UDP *:12817

Eric

[Eric Desrochers (slashd)]

After more investigation, it seems like the 2 extra port-bindings port assignation decision is made by "libdns" itself.

https://github.com/wklaebe/bind9/blob/ac1dcdd124a5abdec4969e2c33836d863bf73aa7/lib/dns/dispatch.c#L1921

[Eric Desrochers (slashd)]

I was able to identify from where the port range list is initialize, and the decision is taken by binary package "libdns162" (source package "bind9")[1] which dhclient relies on.
Basically, it randomly takes any port available between 1024 and 65535[2]

So there is a port randomization mechanism made by the library that take the decision instead of, for instance in some other cases, where the decision is leave it to the kernel[3].

In this particular case both sysctl options :
net.ipv4.ip_local_port_range
net.ipv4.ip_local_reserved_ports

have no effect on port assignation decision.

[1] - https://github.com/wklaebe/bind9/blob/ac1dcdd124a5abdec4969e2c33836d863bf73aa7/lib/dns/dispatch.c#L1921
[2] - https://tools.ietf.org/html/rfc6335
[3] - Example : net.ipv4.ip_local_port_range = 32768 60999

Regards,

[Eric Desrochers (slashd)]

So long story short, starting with Xenial you have the possibility to install "isc-dhcp-client" which doesn't include the 2 extra port bindings, but if you need it for Dynamic DNS (DDNS) you can optionally install "isc-dhcp-client-ddns".

On Trusty, "isc-dhcp-client" combine both.

Will ask the SRU team if it would be acceptable to split up the binary package in two to include changes (already applied in Xenial) to trusty.

[Robie Basak (racb)]

> Will ask the SRU team if it would be acceptable to split up the binary package in two to include changes (already applied in Xenial) to trusty.

I think that it's important that existing Trusty users can continue to use this DDNS functionality after the SRU without any necessary intervention. They may be depending on this functionality. If they have to do something (such as install the additional package manually), then that would be a regression and not acceptable in an SRU.

If there's a way to do the split such that users will automatically get isc-dhcp-client-ddns installed but users bothered by this bug then have the option to switch to the one without it, then that might work. But perhaps it would be simpler to provide users a way to disable this functionality manually? Is there any kind of runtime configuration available?

[Eric Desrochers (slashd)]

> But perhaps it would be simpler to provide users a way to disable this functionality manually? Is there any kind of runtime configuration available?

At runtime, it doesn't seem to have a way, for instance, I already tried the following without success: do-forward-updates false;

# man dhclient.conf
       If you want to do DNS updates in the DHCP client script (see dhclient-script(8)) rather than having the DHCP client do the update directly (for example, if you want to use SIG(0) authentication, which is not supported directly by the DHCP client, you can
       instruct the client not to do the update using the do-forward-updates statement. Flag should be true if you want the DHCP client to do the update, and false if you don't want the DHCP client to do the update. By default, the DHCP client will do the DNS
       update.

To disable the functionality at the moment for Trusty, manipulation in the source package is required.

Undefine NSUPDATE in includes/site.h :

-#define NSUPDATE
+/* #define NSUPDATE */

and disabled failover and tracing in debian/rules :

./configure \
                --prefix=/usr \
                --sysconfdir=/etc/dhcp \
+ --disable-tracing \
+ --disable-failover \

Eric

[Eric Desrochers (slashd)]

I worked on a patch that so far seems to work well.

I still need to arrange a few things, but I should be able to submit a patch soon.

The patch will create a additional binary package.

isc-dhcp-client pkg : dhclient with DDNS functionality disabled (no random extra ports)
isc-dhcp-client-ddns : dhclient with DDNS functionality enabled (with random extra ports)

Eric

[Robie Basak (racb)]

> isc-dhcp-client pkg : dhclient with DDNS functionality disabled (no random extra ports)

What will happen to existing Trusty users who are relying on the current ddns functionality?

[Eric Desrochers (slashd)]

> What will happen to existing Trusty users who are relying on the current ddns functionality?

Existing Trusty user at next isc-dhcp-client pkg upgrade will automatically receive (Recommends: isc-dhcp-client-ddns).

And for users bother by the extra random port they will simply have to apt-get purge isc-dhcp-client-ddns.

Look regression details above for additionnal details.

[Eric Desrochers (slashd)]

I picked "Recommends:" based on following explanation :

Package A recommends Package B, if the package maintainer judges that most users would not want A without also having the functionality provided by B.

Ref:
https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html

Eric

[Robie Basak (racb)]

On Sat, Dec 10, 2016 at 08:20:59PM -0000, Eric Desrochers wrote:
> > What will happen to existing Trusty users who are relying on the
> current ddns functionality?
>
> Existing Trusty user at next isc-dhcp-client pkg upgrade will
> automatically receive (Recommends: isc-dhcp-client-ddns).

Currently (without this SRU), existing Trusty users will have
DDNS-by-default dropped on a release upgrade to Xenial. Users who still
need it would need to install isc-dhcp-client-ddns manually, which is
acceptable in a release upgrade. This is a good thing for security, and
so should continue to happen, right?

If you get all users automatically receiving isc-dhcp-client-ddns in
this SRU, then subsequently the users' behaviour after upgrade to Xenial
will change. They will continue to have isc-dhcp-client-ddns installed,
and so won't get the benefit of this improvement.

> And for users bother by the extra random port they will simply have to
> apt-get purge isc-dhcp-client-ddns.

This part sounds like it'll work well then, but I'm not sure about the
change in release upgrade behaviour above.

In comment #20, you said that there doesn't seem to be a way to disable
DDNS at runtime in Trusty. But this is presumably based on the
upstream-shipped build configuration. How difficult would it be to do by
patching the code?

[Eric Desrochers (slashd)]

> This part sounds like it'll work well then, but I'm not sure about the
change in release upgrade behaviour above.

Good point !

What about "Breaks:" instead of "Recommends:" ?

Package A breaks Package B when both packages cannot be simultaneously configured in a system. The package management system will refuse to install one if the other one is already installed and configured in the system.

> In comment #20, you said that there doesn't seem to be a way to disable
DDNS at runtime in Trusty. But this is presumably based on the
upstream-shipped build configuration. How difficult would it be to do by
patching the code?

Disabling the feature at runtime would definitely be a good alternative. I don't think it will be easily doable without doing some important modification in the src code as AFAIK we can't disable/enable a preprocessor option at runtime.

The C preprocessor modifies a source code file before handing it over to the compiler.

[Eric Desrochers (slashd)]

Rectification :

I think we will need a mix of both world Recommends: and Break: .

Need to check more in deep for implementation though (if applicable)

Recommends: for next pkg upgrade and then rely on Breaks: or something similar to that for following pkg upgrade and release upgrade.

[Eric Desrochers (slashd)]

Hmm need to look more in depth but maybe this would work ? By mentionning the "earlier than" ootion.

* 7.3 Packages which break other packages -

Breaks normally a Breaks entry will have an "earlier than" version clause; such a Breaks is introduced in the version of an (implicit or explicit) dependency which violates an assumption or reveals a bug in earlier versions of the broken pack

Ref:https://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps

[Eric Desrochers (slashd)]

7.6.1 Overwriting files in other packages

It is usually an error for a package to contain files which are on the system in another package. However, if the overwriting package declares that it Replaces the one containing the file being overwritten, then dpkg will replace the file from the old package with that from the new. The file will no longer be listed as "owned" by the old package and will be taken over by the new package. Normally, Breaks should be used in conjunction with Replaces.[53]

For example, if a package foo is split into foo and foo-data starting at version 1.2-3, foo-data would have the fields

     Replaces: foo (<< 1.2-3)
     Breaks: foo (<< 1.2-3)

in its control file. The new version of the package foo would normally have the field

     Depends: foo-data (>= 1.2-3)

(or possibly Recommends or even Suggests if the files moved into foo-data are not required for normal operation)

Ref:
https://www.debian.org/doc/debian-policy/ch-relationships.html

rbasak, what do you think of this ^^ ?

[Eric Desrochers (slashd)]

I will evaluate/test the "debian/control" fields possible scenarios this week, and will update the LP with the outcome.

rbasak, if you have any ideas/suggestion/piece of advise/... I'm all ears.

Eric

[Louis Bouchard (louis)]

Here is how I understand it :

In Trusty, the isc-dhcp client provides the Dynamic DNS (ddns) functionality and this one can be disabled. From Xenial onward, two separate packages provide each of the possibilities.

AFAIU, the Trusty isc-dhcp package should become isc-dhcp which doesn't provide the two extra port bindings, and would also install the isc-dhcp-ddns to provide the extra functionality in order to remain at iso-functionality.

So if isc-dhcp-ddns Breaks/Replaces isc-dhcp and Recommends isc-dhcp-ddns, we should keep the same functionality for existing users. It then becomes possible, by removing isc-dhcp to revoke the use of the two extra port bindings, which is the existing situation on Xenial.

The upgrade path then becomes alligned with the Xenial packaging which has both isc-dhcp & isc-dhcp-ddns.

Am I reading this correctly ?

[Eric Desrochers (slashd)]

@caribou, yes you are reading this correctly.

[Robie Basak (racb)]

I don't follow. isc-dhcp-ddns in Xenial ships /sbin/dhclient and a dpkg-divert in its prerm. I don't see how Breaks/Replaces is relevant. Are you planning something different in Trusty?

Louis said: "...if isc-dhcp-ddns Breaks/Replaces isc-dhcp and Recommends isc-dhcp-ddns...". isc-dhcp-ddns recommending isc-dhcp-ddns doesn't make sense to me. Was this a typo? As I don't follow the Recommends plan, I don't follow this statement.

[Louis Bouchard (louis)]

Notes from online discussion :
==============================
src:isc-dhcp

Trusty:

            4.2.4-7ubuntu12.8 -> (isc-dhcp-client)
            Proposal A: 4.2.4-7ubuntu12.9 -> (isc-dhcp-client, isc-dhcp-client-ddns), isc-dhcp-client Recommends: isc-dhcp-client-ddns
            Proposal B: 4.2.4-7ubuntu12.9 -> (isc-dhcp-client), patched to disable ddns if DISABLE_DDNS_LP_1176046 is set in the environment

Xenial:
            4.3.3-5ubuntu12.6 -> (isc-dhcp-client, isc-dhcp-client-ddns)
            Proposal A: 4.3.3-5ubuntu12.7 -> (isc-dhcp-client, isc-dhcp-client-ddns), isc-dhcp-client Breaks/Replaces: isc-dhcp-ddns (<< 4.3.3-5ubuntu12~)
            Proposal B: no change

Use case 1:

Trusty user has isc-dhcp-client 4.2.4-7ubuntu12.8 installed, and SRUs to end up with 4.2.4-7ubuntu12.9 of both isc-dhcp-client and isc-dhcp-client-ddns.
User upgrades to Xenial.

Expected results: user ends up with isc-dhcp-client installed but not isc-dhcp-client-ddns
Actual results in proposal A: apt resolves this by updating isc-dhcp-client-ddns first to 4.3.3-5ubuntu12.7 and then isc-dhcp-client to 4.3.3-5ubuntu12.7.
Actual results in proposal B: correct.

Use case 2:

Trusty user has isc-dhcp-client 4.2.4-7ubuntu12.8 installed, relies on ddns, and SRUs.
Expected result: ddns support still works.
Actual results in proposal A: correct.
Actual results in proposal B: correct.

Use case 3:

Trusty user has isc-dhcp-client 4.2.4-7ubuntu12.8 installed, ddns is enabled, but this breaks listening on a static UDP port as dhclient sometimes undesirably listens on it first.
Since this is already released, mandatory user intervention is acceptable
Expected result: ddns support is disabled.
Actual results in proposal A: correct with user intervention by manually removing the isc-dhcp-client-ddns package after the SRU.
Actual results in proposal B: correct, with user intervention by adding the environment setting in service startup as a conffile

User then upgrades to Xenial.
Expected results: the lack of DDNS support is still maintained.
Actual results in proposal B: correct: the environment variable is ignored.

[Robie Basak (racb)]

> Actual results in proposal A: apt resolves this by updating isc-dhcp-client-ddns first to 4.3.3-5ubuntu12.7 and then isc-dhcp-client to 4.3.3-5ubuntu12.7.

To be clear, that means I think the behaviour in proposal A would be wrong. I'm open to further suggestions on what we could do, or why my reasoning is wrong or irrelevant.

[Eric Desrochers (slashd)]

Maybe we are not looking the situation the right way ... What if we look at this the opposite way ?

"isc-dhcp-client" remains the same including the extra random bindings ports.

and we introduce a new package "isc-dhcp-client-noddns" that change the dhclient behavior by overriding dhclient and introduce the disable DDNS feature version of dhclient without the extra random bindings ports.

Assuming that most user will stick with "isc-dhcp-client", then the upgrade will be smooth.

# Use case #1
User that stick with "isc-dhcp-client" that relies on DDNS functionality at release upgrade will have to manually install "isc-dhcp-client-ddns" to have the same functionality as "isc-dhcp-client" in Trusty.

From my understanding, the above is an acceptable behavior ? Please correct me if I'm wrong.

# Use case #2
User that decide to install "isc-dhcp-client-noddns" mean they don't relies on DDNS functionnality, so at next release upgrade, they will get "isc-dhcp-client" and won't notice any change on their side. And if one day they need the DDNS feature, then they will simply have to manually install the "isc-dhcp-client-ddns" package.

Not sure on how feasible it is to introduce a package that will no longer exist in next release upgrade, maybe someone can shed some light on this if it's doable or not.

Does that make sense ?

Eric

[Eric Desrochers (slashd)]

@caribou
@rbasak

Let me share the work (since rbasak seems to like the approach) I've done so far on my above recent proposal :

isc-dhcp-client : dhclient with DDNS functionality enabled
isc-dhcp-client-noddns : dhclient with DDNS functionality disabled

This is a working debdiff, I think the only things that need to be covered is the release upgrade portion.

Let me know what you think so far.

[Robie Basak (racb)]

Let me try and restate this in detail to make sure I understand it.

Trusty: add a new binary package called isc-dhcp-client-noddns. This will work the same as isc-dhcp-client-ddns in Xenial, but in reverse. It will use the same dpkg-divert mechanism to provide a replacement /sbin/dhclient that does not have the DDNS behaviour. Users must opt in to it by installing this new package, so there should be no change in behaviour to existing Trusty users and thus low regression risk.

Xenial: add a isc-dhcp-client-noddns transitional binary package that deactivates the dpkg-divert and depends on isc-dhcp-client. "Default" users upgrading from Trusty to Xenial are unaffected since they would never have -noddns installed. Trusty users who had opted in to isc-dhcp-client-noddns will transition to the Xenial isc-dhcp-client package that provides no DDNS. Trusty users who are using isc-dhcp-client and want to keep the DDNS support need to install isc-dhcp-client-ddns in Xenial, as before. So no change in behaviour for any use case.

One thing that will need testing is upgrade ordering from Trusty to Xenial with respect to the divert. For example, what if I am using isc-dhcp-client-noddns and ask apt to upgrade me and add isc-dhcp-client-ddns at the same time? Will the overlapping dpkg-diverts behave themselves?

I don't see any problem with this approach, though it does require new binaries in both Trusty and Xenial, which is a little ugly especially for Xenial as there isn't a bug in Xenial currently. I'd appreciate input from another SRU team member on this proposal.

I haven't reviewed your debdiff in detail - let's get general agreement that this is a sensible approach first to avoid any wasted effort. But I did notice:

> +++ isc-dhcp-4.2.4/debian/changelog
> + isc-dhcp-client : dhclient with DDNS functionality enabled

Perhaps add "(no behavioural change from previous version)" or something, to make it clear to existing users seeing the SRU changelog that they don't need to panic? Just noting for the future; no need to do anything now. I think we should get an ack on the approach from ~ubuntu-sru before proceeding further.

[Eric Desrochers (slashd)]

@rbasak,

Yes, right this is exactly what I had in mind.

Noted for "(no behavioural change from previous version)".

[Eric Desrochers (slashd)]

Here a new version of the actual in progress... debdiff for Trusty.

[Adam Conrad (adconrad)]

Third option: Tell users who are bothered by this misfeature to upgrade to Xenial. "I don't like these ports being in use" isn't a bug, per se, and our goal should not be to backport exact feature parity from Xenial to Trusty. We release new releases for a variety of reasons, and this is one of them.

I understand the compulsion to always make every LTS "look and behave the same", but there are points of diminishing returns, and this is one of them. To quote pitti (on another backporting topic), "if this is really what we want, why don't we just symlink trusty to xenial?"

If there was a bug to be fixed here, sure, let's fix it, but I don't see one worth fixing. Long-time trusty users have been at this for over 2.5 years, and I don't see a flood of people complaining about these random ports, while Xenial has been out for over half a year now, and the few people genuinely irritated can, well, use Xenial.

[Eric Desrochers (slashd)]

Installing isc-dhcp-client-noddns on Trusty and then upgrade to Xenial with a local archive of a modified Xenial's isc-dhcp-client to replace isc-dhcp-client-noddns package to isc-dhcp-client during release upgrade from Trusty to Xenial work as expected with a completed successful release upgrade.

+Replaces:
+ isc-dhcp-client-noddns (<< 4.2.4-7ubuntu12.8+noddnsbuild1),

Capture after upgrade :

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

# dpkg -l | grep -i dhcp
ii isc-dhcp-client 4.3.3-5ubuntu12.7 amd64 DHCP client for automatically obtaining an IP address
rc isc-dhcp-client-noddns 4.2.4-7ubuntu12.8+noddnsbuild1 amd64 Dynamic DNS (DDNS) enabled DHCP client
ii isc-dhcp-common 4.3.3-5ubuntu12.6 amd64 common files used by all of the isc-dhcp packages

# lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 707 root 6u IPv4 16437 0t0 UDP *:bootpc

# ls -altr /sbin/dhclient*
-rwxr-xr-x 1 root root 15219 Jan 11 09:12 /sbin/dhclient-script
-rwxr-xr-x 1 root root 487248 Jan 11 09:12 /sbin/dhclient

The only thing that could be improve that I can't think of right now, is to not make isc-dhcp-client-noddns not turn into "rc" state by making sure its configuration files doesn't remain.

- Eric

[Eric Desrochers (slashd)]

Quick note saying that the work is still in progress....

[Eric Desrochers (slashd)]

Feedback from the field about the test package of "isc-dhcp-client-noddns" I made for someone interested to give it a try before the SRU:

"... we've tried it out and didn’t notice anything unusual, but I suspect we would only see problems if there are any if we ran it 1000s of times in ci.

I also don’t see any additional random ports consumed which is good:

# lsof -p 485
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dhclient 485 root cwd DIR 202,1 4096 2 /
dhclient 485 root rtd DIR 202,1 4096 2 /
dhclient 485 root txt REG 202,1 7543253 9727 /sbin/dhclient
dhclient 485 root mem REG 202,1 43616 8991 /lib/x86_64-linux-gnu/libnss_files-2.19.so
dhclient 485 root mem REG 202,1 1840928 8950 /lib/x86_64-linux-gnu/libc-2.19.so
dhclient 485 root mem REG 202,1 149120 8938 /lib/x86_64-linux-gnu/ld-2.19.so
dhclient 485 root 0u CHR 1,3 0t0 6 /dev/null
dhclient 485 root 1u CHR 1,3 0t0 6 /dev/null
dhclient 485 root 2u CHR 1,3 0t0 6 /dev/null
dhclient 485 root 3w REG 202,1 1461 135467 /var/lib/dhcp/dhclient.eth0.leases
dhclient 485 root 4u pack 9509 0t0 ALL type=SOCK_RAW
dhclient 485 root 5u IPv4 9512 0t0 UDP *:bootpc
dhclient 485 root 6u unix 0xffff8800e7069000 0t0 15863 socket
"

[Robie Basak (racb)]

14:45 <rbasak> slashd: I'm sure I asked you this before, and you answered, but I can't find your answer anywhere, sorry. Are you proposing your debdiff in commit 40 as ready for upload to Trusty? Or is it still in progress? And do you have a debdiff for the transitional package in Xenial please? That needs to go in first.

[Eric Desrochers (slashd)]

Hi Robie,

To answer your questions...

* Are you proposing your debdiff in commit 40 as ready for upload to Trusty?
No, here's the final one (that I sent you by email), but I will attach them to the LP bug now.

verif-trusty.debdiff
verif-xenial.debdiff

* Or is it still in progress?
No, I did both debdiffs, and I tested it with differents scenarios (my note are above in the LP bug)
caribou also did some testing (upgrade Trusty->Xenial,...) and didn't see any particular problems.

Additionally, I have a users running the new binary package from a more than a month, and everything goes well and as expected (without the random extra port).

* And do you have a debdiff for the transitional package in Xenial please?
Yes, verif-xenial.debdiff

Eric

[Eric Desrochers (slashd)]

@rbasak,

* Un-version the "Replaces:" as discussed : DONE
- Upgrade Trusty->Xenial with isc-client-dhcp pkg successfully completed with expected result
- Upgrade Trusty->Xenial with isc-client-dhcp-noddns pkg successfully completed with expected result

* Fix typo here and there : DONE

* Add (LP: #1176046) in X changelog : DONE

* Update SRU information : DONE

Eric

[Norman Henderson (norm-audrey)]

Eric, please see 1670303 - pending a substantive solution there, would you be willing to fix dhcpd in the corresponding way that you have fixed dhclient here, i.e. by making separate packages available with or without ddns support?

[Eric Desrochers (slashd)]

@Norman Henderson (norm-audrey)

I would suggest you to report a separate bug/wishlist for this topic.

Thanks

[Robie Basak (racb)]

Hi Eric,

Sorry for the delay in getting back to this. As it has been a while I have reviewed everything again from the beginning.

Can I confirm that by "with expected result" you mean that on Trusty->Xenial upgrade, in both cases (isc-client-dhcp installed and isc-client-dhcp-noddns installed) you end up with isc-client-dhcp installed in Xenial?

The latest debdiffs in comments 51 and 53 look fine to me - please go ahead and upload.

For SRU verification, please could you make sure you test each upgrade path permutation again, and before upload, just update the SRU information to confirm that you will be doing this? Thanks!

[Eric Desrochers (slashd)]

Hi Robie,

Thanks for your review/feedbacks.

At the moment, there is currently an SRU for isc-dhcp via LP: #1689854 for Trusty.

As soon as this SRU is successfully completed. (move to -updates). I'll start the SRU for this particular bug.

- Eric

[Eric Desrochers (slashd)]

> Can I confirm that by "with expected result" you mean that on Trusty->Xenial upgrade, in both cases (isc-client-dhcp installed and isc-client-dhcp-noddns installed) you end up with isc-client-dhcp installed in Xenial?

yes.

[Robie Basak (racb)]

Hello Rodney, or anyone else affected,

Accepted isc-dhcp into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/isc-dhcp/4.2.4-7ubuntu12.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Rodney, or anyone else affected,

Accepted isc-dhcp into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/isc-dhcp/4.3.3-5ubuntu12.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Eric Desrochers (slashd)]

Dear Rodney, and anyone else affected,

Please install "isc-dhcp-client-noddns" found in trusty-proposed and provide feedbacks by commenting the bug. Your feedbacks are paramount for the success of this SRU.

Additionally, I will also do some precision testing on Trusty and re-run the release upgrade scenarios from Trusty to Xenial.

Eric

[David Coronel (davecore)]

I installed isc-dhcp-client-noddns from trusty-proposed and I don't get the two random extra high ports anymore:

Before:

# dpkg -l | grep isc-dhcp-
ii isc-dhcp-client 4.2.4-7ubuntu12.9 amd64 ISC DHCP client
ii isc-dhcp-common 4.2.4-7ubuntu12.9 amd64 common files used by all the isc-dhcp* packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 716/dhclient
udp 0 0 0.0.0.0:42496 0.0.0.0:* 716/dhclient
udp6 0 0 :::7781 :::* 716/dhclient

After:

# dpkg -l | grep isc-dhcp-
ii isc-dhcp-client 4.2.4-7ubuntu12.10 amd64 ISC DHCP client
ii isc-dhcp-client-noddns 4.2.4-7ubuntu12.10 amd64 Dynamic DNS (DDNS) disabled DHCP client
ii isc-dhcp-common 4.2.4-7ubuntu12.10 amd64 common files used by all the isc-dhcp* packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 13411/dhclient

And if I remove the isc-dhcp-client-noddns package, I get the 2 random ports again:

# dpkg -l | grep isc-dhcp-
ii isc-dhcp-client 4.2.4-7ubuntu12.10 amd64 ISC DHCP client
ii isc-dhcp-common 4.2.4-7ubuntu12.10 amd64 common files used by all the isc-dhcp* packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 565/dhclient
udp 0 0 0.0.0.0:63510 0.0.0.0:* 565/dhclient
udp6 0 0 :::44100 :::* 565/dhclient

Looks good to me.

[Rodney Beede (business2008+launchpad)]

Verified works on trusty as desired. No more extra ports.

[Rodney Beede (business2008+launchpad)]

Can confirm for xenial installing the isc-dhcp-client-ddns also works as expected. This package enables ddns ports.

When not using this package no random ddns ports as desired.

[Eric Desrochers (slashd)]

[Verification Trusty]

ii isc-dhcp-client-noddns 4.2.4-7ubuntu12.10 amd64 Dynamic DNS (DDNS) disabled DHCP client

# netstat -anputa | grep dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 1676/dhclient

# ls -altr /sbin/dhclient*
-rwxr-xr-x 1 root root 7545313 May 25 12:30 /sbin/dhclient
-rwxr-xr-x 1 root root 15286 May 25 12:32 /sbin/dhclient-script
-rwxr-xr-x 1 root root 1668160 May 25 12:32 /sbin/dhclient-ddns # Binary from pkg "isc-dhcp-client"

-----

ii isc-dhcp-client 4.2.4-7ubuntu12.10 amd64 ISC DHCP client

# netstat -anputa | grep dhclient
udp 0 0 0.0.0.0:7712 0.0.0.0:* 1685/dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 1685/dhclient
udp6 0 0 :::36532 :::* 1685/dhclient

# ls -altr /sbin/dhclient*
-rwxr-xr-x 1 root root 15286 May 25 12:32 /sbin/dhclient-script
-rwxr-xr-x 1 root root 1668160 May 25 12:32 /sbin/dhclient

No behavioural changes for "isc-dhcp-client", as expected

[Eric Desrochers (slashd)]

[Verification Xenial]

{Before upgrade}
===
ii isc-dhcp-client 4.2.4-7ubuntu12.10 amd64 ISC DHCP client
ii isc-dhcp-client-noddns 4.2.4-7ubuntu12.10 amd64 Dynamic DNS (DDNS) disabled DHCP client
ii isc-dhcp-common 4.2.4-7ubuntu12.10 amd64 common files used by all the isc-dhcp* packages

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 1686/dhclient

# ls -altr /sbin/dhclient*
-rwxr-xr-x 1 root root 7545313 May 25 12:30 /sbin/dhclient
-rwxr-xr-x 1 root root 15286 May 25 12:32 /sbin/dhclient-script
-rwxr-xr-x 1 root root 1668160 May 25 12:32 /sbin/dhclient-ddns

# more /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

{After upgrade}
===
ii isc-dhcp-client 4.3.3-5ubuntu12.7 amd64 DHCP client for automatically obtaining an IP address

# netstat -anputa | grep -i dhclient
udp 0 0 0.0.0.0:68 0.0.0.0:* 150/dhclient

^^ expected result since Xenial does the opposite :
isc-dhcp-client : with random extra ports
isc-dhcp-client-ddns : with random extra ports

Basically, Xenial users has to install "isc-dhcp-client-ddns" if they want the DDNS functionality.

# dist-upgrade/history.log
Remove: libxapian22:amd64 (1.2.16-2ubuntu1), ==> isc-dhcp-client-noddns:amd64 (4.2.4-7ubuntu12.10) <==, systemd-services:amd64 (204-5ubuntu20.24), python3.4:amd64 (3.4.3-1ubuntu1~14.04.5), python3.4-minimal:amd64 (3.4.3-1ubuntu1~14.04.5), libpython3.4-stdlib:amd64 (3.4.3-1ubuntu1~14.04.5), libpython3.4-minimal:amd64 (3.4.3-1ubuntu1~14.04.5), perl-modules:amd64 (5.18.2-2ubuntu1.1), aptitude:amd64 (0.6.8.2-1ubuntu4), libasprintf0c2:amd64 (0.18.3.1-1ubuntu3)
End-Date: 2017-05-29 11:00:57

Package has been removed :
rc isc-dhcp-client-noddns 4.2.4-7ubuntu12.10 amd64 Dynamic DNS (DDNS) disabled DHCP client

The upgrade does remove if installed "isc-dhcp-client-noddns" and replaces it by "isc-dhcp-client".

[Launchpad Janitor (janitor)]

This bug was fixed in the package isc-dhcp - 4.2.4-7ubuntu12.10

---------------
isc-dhcp (4.2.4-7ubuntu12.10) trusty; urgency=medium

  * Introduction of a new binary package "isc-dhcp-client-noddns" (LP: #1176046)
    - isc-dhcp-client : no behavioural change from previous version.
    - isc-dhcp-client-noddns : dhclient with DDNS functionality disabled.

    The current functionality in Trusty is to provide DDNS to dhclient by
    default. Note that, this is the opposite in Xenial where dhclient,
    by default, offers no DDNS functionality.

  * Disable tracing and failover.

    The tracing and failover are stricly disabled for the new package
    "isc-dhcp-client-noddns" and doesn't apply to package "isc-dhcp-client".

    In order to successfully build "isc-dhcp-client-noddns" without
    -DNSUPDATE and stay align with upstream Debian git commit [0bbf7c7]
    (closes: #712503) which implies the following:

    - debian/rules : Disable support for failover
      (DHCP Failover is a mechanism whereby two DHCP servers are both configured
      to manage the same pool of addresses so that they can share the load of
      assigning leases for that pool, and provide back-up for each other in case
      of network outages.
      (Define to include Failover Protocol support)

    - debian/rules : Disable support for server activity tracing
      (Server tracing support)

  * CFLAGS configuration change
    - debian/rules : Add backslash '\' characters for CLAGS options :
      "-D_PATH_DHCLIENT_CONF" , "-D_PATH_DHCPD_CONF" , "-D_PATH_DHCLIENT_SCRIPT"
      to avoid compilation errors expression before '/' token is expected at
      compile time.
    - debian/rules : Remove "export CLAGS" line, since it is now passing during
      ./configure in order to control when compiling with or without DDNS
      functionality.

 -- Eric Desrochers <email address hidden> Wed, 24 May 2017 08:41:07 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for isc-dhcp has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu12.7

---------------
isc-dhcp (4.3.3-5ubuntu12.7) xenial; urgency=medium

  * debian/control : Add "Replaces:" option for package isc-dhcp-client
    to allow the packaging system to remove "isc-dhcp-client-noddns" on
    systems on which it is installed and replace it by "isc-dhcp-client"
    during a release upgrade from Trusty to Xenial. (LP: #1176046)

 -- Eric Desrochers <email address hidden> Wed, 01 Feb 2017 08:36:18 -0500