dhclient -6: Can't bind to dhcp address: Cannot assign requested address

Bug #1447715 reported by Eloy Paris
106
This bug affects 23 people
Affects Status Importance Assigned to Milestone
ifupdown (Debian)
Fix Released
Unknown
ifupdown (Ubuntu)
Fix Released
Medium
Dan Streetman
Xenial
Fix Released
Medium
Dan Streetman

Bug Description

[Impact]

When using dhcpv6 configured via ifupdown, the interface's dhcp client fails to start at boot with the error:

Can't bind to dhcp address: Cannot assign requested address

This is because ifupdown doesn't wait, after bringing the interface up, for it to complete its link-local Duplicate Address Detection (DAD), and the dhcp client can't bind to the link-local address because it's still in "tentative" state (until DAD completes).

This is fixed upstream in debian ifupdown in version 0.8.11 by adding '-tentative' to the /lib/ifupdown/wait-for-ll6.sh script; before the script was only waiting for the link-local address to appear, now it waits until the link-local address appears and is not in 'tentative' state.

[Test Case]

Original test case, before dhclient was fixed:
Configure an interface, using ifupdown, with dhcpv6 (e.g. iface eth0 inet6 dhcp) and reboot. It will fail to get a dhcp address during boot.

New test case, showing failure when using dhclient in 'stateless' mode:
Configure ifupdown like:

auto eth0
iface eth0 inet6 auto
  dhcp 1

When eth0 is down, run 'sudo ifup eth0', and it will fail as dhclient fails to start on the interface.

[Regression Potential]

As this modifies the wait-for-dad function, its potential for regression is to introduce failures in starting dhclient, possibly by failing to wait for DAD to complete or due to some other failure in the wait for dad function.

[Other Info]

After upgrading to Ubuntu 15.04 my computer is unable to obtain an IPv6 address via DHCPv6. The root cause is that dhclient is exiting with the following message:

-----------------------------------------------------
Can't bind to dhcp address: Cannot assign requested address
Please make sure there is no other dhcp server
running and that there's no entry for dhcp or
bootp in /etc/inetd.conf. Also make sure you
are not running HP JetAdmin software, which
includes a bootp server.

If you think you have received this message due to a bug rather
than a configuration issue please read the section on submitting
bugs on either our web page at www.isc.org or in the README file
before submitting a bug. These pages explain the proper
process and the information we find helpful for debugging..

exiting.
-----------------------------------------------------

The problem seems to exist in isc-dhcp-client 4.3.1-5ubuntu2 (15.04 version) because downgrading to isc-dhcp-client 4.2.4-7ubuntu14 (14.10 version) allows me to obtain an address.

NetworkManager is the one launching dhclient. The two invocations (one for IPv4 and one for IPv6) are:

dhclient_start(): running: /sbin/dhclient -d -q -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /run/sendsigs.omit.d/network-manager.dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-c0a3dfde-5c0b-4cef-9c3b-563cfb1fb9d2-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0

dhclient_start(): running: /sbin/dhclient -d -q -6 -N -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /run/sendsigs.omit.d/network-manager.dhclient6-eth0.pid -lf /var/lib/NetworkManager/dhclient6-c0a3dfde-5c0b-4cef-9c3b-563cfb1fb9d2-eth0.lease -cf /var/lib/NetworkManager/dhclient6-eth0.conf eth0

I do not see anything suspicious with the way both dhclients are invoked.

Given that a downgrade allows me to obtain an IPv6 address I think this might be a bug in the ISC DHCP client rather than in NetworkManager.

Related Bugs:
 * bug 1633479: dhclient does not wait for ipv6 dad (duplicate address detection)

Revision history for this message
Eloy Paris (peloy-chapus) wrote :

A quick update -- This seems to be a timing issue, i.e. dhclient trying to bind to the DHCPv6 port (UDP 546) on a link-local address when the interface is not ready, but I am not sure. What I do know is that the following patch to isc-dhcp-client 4.3.1 works around the problem:

--- a/common/socket.c
+++ b/common/socket.c
@@ -170,6 +170,11 @@
   if ((info->flags & INTERFACE_STREAMS) == INTERFACE_UPSTREAM) {
    *do_multicast = 0;
   }
+
+ /*
+ * Wait a little before calling bind(). See LP #1447715.
+ */
+ sleep(5);
   break;
 #endif /* DHCPv6 */

With this patch dhclient -6 waits 5 seconds before attempting to bind(), and that allows me to obtain an IPv6 address from the DHCPv6 server.

A machine running ArchLinux and network-manager 1.0 does not have this problem of dhclient quitting after failing to bind(). Ubuntu 15.04 uses network-manager 0.9.10.0 so this might be a problem with Network Manager after all. We will have to wait and see until Ubuntu has Network Manager 1.0. In the meantime, I am all see with the above "workaround".

Cheers,

Eloy Paris.-

Revision history for this message
Seth Arnold (seth-arnold) wrote :

I understand that binding to ipv6 addresses that haven't finished Duplicate Address Detection causes the bind() to fail.

http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ suggests using IP_FREEBIND:

       IP_FREEBIND (since Linux 2.4)
              If enabled, this boolean option allows binding to an IP
              address that is nonlocal or does not (yet) exist. This
              permits listening on a socket, without requiring the
              underlying network interface or the specified dynamic IP
              address to be up at the time that the application is trying to
              bind to it. This option is the per-socket equivalent of the
              ip_nonlocal_bind /proc interface described below.

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu):
status: New → Confirmed
Revision history for this message
James Johnston (mail-codenest) wrote :

I can confirm that IPv6 on Ubuntu 15.10 is basically broken out of the box... at least, the DHCPv6 is broken out of the box due to this issue. The workaround of setting disable_dad worked for me, described at https://www.agwa.name/blog/post/beware_the_ipv6_dad_race_condition

Revision history for this message
James Johnston (mail-codenest) wrote :

OK, the last comment was a bit premature. After rebooting another time or two, it seems that disable_dad did NOT help for me. Still happens sometimes on boot.

Revision history for this message
James Johnston (mail-codenest) wrote :

But this worked:

iface enp0s3 inet6 auto
iface enp0s3 inet6 dhcp
    pre-up sleep 10

But a gross ugly hack...

Revision history for this message
Kevin Otte (nivex) wrote :

This is still an issue in Ubuntu Server 16.04.1. I have the following in my /etc/network/interfaces file:

# The primary network interface
auto ens3
iface ens3 inet dhcp
# This is an autoconfigured IPv6 interface
iface ens3 inet6 auto
    dhcp 1

And on startup:
Aug 5 15:55:23 flounder dhclient[2483]: Can't bind to dhcp address: Cannot assign requested address

Can confirm the "gross ugly hack" in #7 works. I suggest involving the ifupdown and NetworkManager maintainers as well since their tools should probably be aware of the state of the link and can control when dhclient is invoked.

Changed in dhcp:
status: Unknown → New
Revision history for this message
LaMont Jones (lamont) wrote :

Better yet, have dhclient retry that bind call itself, at least for a short while, before throwing an error.

Revision history for this message
Dan Streetman (ddstreet) wrote :

This is fixed in debian ifupdown 0.8.11 by this patch:
--- ifupdown-0.8.10/wait-for-ll6.sh 2015-12-01 16:50:26.000000000 -0500
+++ ifupdown-0.8.11/wait-for-ll6.sh 2016-04-20 08:57:37.000000000 -0400
@@ -4,7 +4,7 @@
 delay=${IF_LL_INTERVAL:-0.1}

 for attempt in $(seq 1 $attempts); do
- lladdress=$(ip -6 -o a s dev "$IFACE" scope link)
+ lladdress=$(ip -6 -o a s dev "$IFACE" scope link -tentative)
  if [ -n "$lladdress" ]; then
   attempt=0
   break

Changed in isc-dhcp (Ubuntu):
status: Confirmed → Invalid
no longer affects: isc-dhcp (Ubuntu)
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1447715-xenial.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Dan Streetman (ddstreet) wrote :

This is also fixed in network-manager upstream (git://git.freedesktop.org/git/NetworkManager/NetworkManager.git) by commit d1295b12e9f802867edef57ee02c87495df1683e ("dhcp: wait for an IPv6LL address before starting DHCPv6 (bgo #740147)") which is included in Ubuntu network-manager starting in wily.

Changed in ifupdown (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet)
description: updated
Dan Streetman (ddstreet)
Changed in ifupdown (Ubuntu Xenial):
assignee: nobody → Dan Streetman (ddstreet)
Changed in ifupdown (Ubuntu Precise):
assignee: nobody → Dan Streetman (ddstreet)
Changed in ifupdown (Ubuntu Trusty):
assignee: nobody → Dan Streetman (ddstreet)
Changed in ifupdown (Ubuntu):
status: New → Fix Released
affects: dhcp → ifupdown
Changed in ifupdown (Ubuntu Precise):
status: New → Invalid
Changed in ifupdown (Ubuntu Xenial):
status: New → In Progress
Changed in ifupdown (Ubuntu Trusty):
status: New → In Progress
importance: Undecided → Medium
Changed in ifupdown (Ubuntu Xenial):
importance: Undecided → Medium
Changed in ifupdown (Ubuntu):
importance: Undecided → Medium
Changed in ifupdown (Ubuntu Precise):
importance: Undecided → Medium
Revision history for this message
Dan Streetman (ddstreet) wrote :

I tested on precise and trusty, and this problem doesn't appear there, so only xenial requires the ifupdown SRU.

Changed in ifupdown (Ubuntu Trusty):
status: In Progress → Invalid
Revision history for this message
Dan Streetman (ddstreet) wrote :

And yakkety has ifupdown 0.8.13 which contains the fix already.

tags: added: sts-sru
Changed in ifupdown:
status: New → Fix Released
Mathew Hodson (mhodson)
affects: ifupdown → ubuntu-translations
Changed in ubuntu-translations:
importance: Unknown → Undecided
status: Fix Released → New
no longer affects: ubuntu-translations
no longer affects: ifupdown (Ubuntu Trusty)
no longer affects: ifupdown (Ubuntu Precise)
description: updated
Changed in ifupdown (Debian):
status: Unknown → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

I found this bug separately and filed bug 1633479.
The fix i intended was to dhclient, so that dhclient would wait in PREINIT6 dhclient-script.

I think that is generally more functional as it would mean ifupdown or any other user of dhclient would get the fix.

Thoughts?

Revision history for this message
Dan Streetman (ddstreet) wrote :

> The fix i intended was to dhclient, so that dhclient would wait in PREINIT6 dhclient-script.

Looks good to me, fixing dhclient as well is a good idea.

> I think that is generally more functional as it would mean ifupdown or any other user of dhclient
> would get the fix.

What other users are there of dhclient, besides ifupdown (and manually running dhclient)?

Revision history for this message
Scott Moser (smoser) wrote :

Dan,
The use case I found was in initramfs-tools, which recently SRU'd a change that uses dhclient to support 'ip=' command lines and dhcpv6.

Revision history for this message
Dan Streetman (ddstreet) wrote :

> The use case I found was in initramfs-tools, which recently SRU'd a change that uses dhclient to
> support 'ip=' command lines and dhcpv6.

aha, and busybox ip doesn't support the -tentative param that the ifupdown change uses.

+1 on fixing isc-dhcp then (as well as leaving this fix in ifupdown).

Dan Streetman (ddstreet)
tags: added: sts-sponsor
Revision history for this message
Dan Streetman (ddstreet) wrote :

update for this:

this patch is already included upstream in debian, and is already included in yakkety. It's not yet in xenial. It's not required in trusty (bug doesn't exist there).

additionally, bug 1633479 (which fixes this by putting the DAD wait in isc-dhcp-client) is in proposed for p/t/x/y. The root problem can be fixed in either place (ifupdown via this bug, or isc-dhcp-client via that bug), and doesn't need to be fixed in both.

So SRUing this to xenial will match the code that's upstream, and will fix the bug for systems that don't have an updated isc-dhcp-client (though I don't know why anyone would update ifupdown but not isc-dhcp-client). But, once bug 1633479 is moved from proposed to released in xenial, this SRU isn't technically necessary, other than to keep code/behavior similar between xenial and y+ ifupdown.

Revision history for this message
Martin Pitt (pitti) wrote :

bug 1633479 actually was released to -updates now, so is there still actually some observable behaviour difference that can be used to both justify and verify this update?

Scott Moser (smoser)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

Martin, I'm not sure.
I just realized that the original report shows a dhclient line of

/sbin/dhclient -d -q -6 -N -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /run/sendsigs.omit.d/network-manager.dhclient6-eth0.pid -lf /var/lib/NetworkManager/dhclient6-c0a3dfde-5c0b-4cef-9c3b-563cfb1fb9d2-eth0.lease -cf /var/lib/NetworkManager/dhclient6-eth0.conf eth0

the '-sf' is scriptfile, which defaults to dhclient-script, which is where the patch in bug 1633479 was made. The fix proposed here was to ifupdown, but it would seem to me that since network manager provides its own script-file, that NetworkManager is still not fixed, unless it somehow uses /sbin/dhclient-script (it might)

Revision history for this message
Dan Streetman (ddstreet) wrote :

for NetworkManager, this is fixed starting in wily (see comment 13) and so should work in xenial/yakkety; and in trusty, dhclient still works (per description stating dhclient works in isc-dhcp-client 4.2.4-7ubuntu14). I'm not sure if NetworkManager using dhcpv6 works in precise (but who is using precise with NetworkManager?). It would be useful if anyone could test NetworkManager on precise using dhcpv6.

for ifupdown/dhclient, as scott said in bug 1633479 comment 13, in older trusty/precise dhclient, it actually retries and so the bug doesn't exist there. In yakkety, this is already fixed in ifupdown (via fix included in debian ifupdown pulled into yakkety). So this bug only is possibly applicable in xenial; and I just confirmed that by updating only isc-dhcp-client to version 4.3.3-5ubuntu12.4, it does succeed in getting a dhcpv6 address.

So no, it doesn't seem that this patch to ifupdown is needed in xenial, and if applied I don't think there would be any noticable difference. So it should be fine to mark this Won't Fix for xenial, as it was fixed in isc-dhcp-client via bug 1633479.

Revision history for this message
Michael Terry (mterry) wrote :

Thank you Dan, for the analysis. Per comment #23, I'll mark Won't Fix for xenial.

Changed in ifupdown (Ubuntu Xenial):
status: In Progress → Won't Fix
Louis Bouchard (louis)
tags: added: sts-sru-done
removed: sts-sru
Revision history for this message
Dan Streetman (ddstreet) wrote :

I'm re-opening this for Xenial, moving from Wontfix back to in progress. This is because while 'normal' dhcpv6 works without this fix because of the patch to dhclient (see my explanation in comment 23), there is a special case of using dhclient in 'stateless' mode, i.e. dhclient -6 -S. In that mode, dhclient does not call its 'PREINIT6' which is where the wait-for-dad patch was added. When configuring ifupdown in this way, dhclient still fails due to this bug; e.g.:

auto eth0
iface eth0 inet6 auto
  dhcp 1

if eth0 is down, then running ifup eth0 with the above config will result in dhclient always failing due to this bug.

Changed in ifupdown (Ubuntu Xenial):
status: Won't Fix → In Progress
Revision history for this message
Dan Streetman (ddstreet) wrote :

i opened bug 1764478 to change dhclient 'stateless' mode, but as explained in that bug, it's better to fix ifupdown, here.

Revision history for this message
Dan Streetman (ddstreet) wrote :
Revision history for this message
Dan Streetman (ddstreet) wrote :
Dan Streetman (ddstreet)
description: updated
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Eloy, or anyone else affected,

Accepted ifupdown into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.8.10ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ifupdown (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Dan Streetman (ddstreet) wrote :
Download full text (3.2 KiB)

Xenial verification:

ubuntu@lp1447715:/etc/network/interfaces.d$ dpkg -l | grep ifupdown
ii ifupdown 0.8.10ubuntu1.2 amd64 high level tools to configure network interfaces

ubuntu@lp1447715:/etc/network/interfaces.d$ grep -A 2 ens7 50-cloud-init.cfg
auto ens7
iface ens7 inet6 auto
 dhcp 1

ubuntu@lp1447715:/etc/network/interfaces.d$ sudo ip l show ens7
3: ens7: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:b0:86:a5 brd ff:ff:ff:ff:ff:ff

ubuntu@lp1447715:/etc/network/interfaces.d$ sudo ifup -v ens7
Parsing file /etc/network/interfaces.d/50-cloud-init.cfg
Configuring interface ens7=ens7 (inet6)
/bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/ethtool
run-parts: executing /etc/network/if-pre-up.d/ifenslave
+ [ inet6 = meta ]
+ IF_BOND_SLAVES=
+ [ ]
+ [ ]
+ [ -z ]
+ exit
run-parts: executing /etc/network/if-pre-up.d/vlan
/sbin/modprobe -q net-pf-10 > /dev/null 2>&1 || true # ignore failure.

/sbin/sysctl -q -e -w net.ipv6.conf.ens7.accept_ra=2
/sbin/sysctl -q -e -w net.ipv6.conf.ens7.autoconf=1
/bin/ip link set dev ens7 up
/lib/ifupdown/wait-for-ll6.sh
/sbin/dhclient -1 -6 -S -pf /run/dhclient6.ens7.pid -lf /var/lib/dhcp/dhclient6.ens7.leases -I -df /var/lib/dhcp/dhclient.ens7.leases ens7
Failed to bring up ens7.

ubuntu@lp1447715:/etc/network/interfaces.d$ dpkg -l | grep ifupdown
ii ifupdown 0.8.10ubuntu1.3 amd64 high level tools to configure network interfaces

ubuntu@lp1447715:/etc/network/interfaces.d$ grep -A 2 ens7 50-cloud-init.cfg
auto ens7
iface ens7 inet6 auto
 dhcp 1

ubuntu@lp1447715:/etc/network/interfaces.d$ sudo ip l show ens7
3: ens7: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:b0:86:a5 brd ff:ff:ff:ff:ff:ff

ubuntu@lp1447715:/etc/network/interfaces.d$ sudo ifup -v ens7
Parsing file /etc/network/interfaces.d/50-cloud-init.cfg
Configuring interface ens7=ens7 (inet6)
/bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/ethtool
run-parts: executing /etc/network/if-pre-up.d/ifenslave
+ [ inet6 = meta ]
+ IF_BOND_SLAVES=
+ [ ]
+ [ ]
+ [ -z ]
+ exit
run-parts: executing /etc/network/if-pre-up.d/vlan
/sbin/modprobe -q net-pf-10 > /dev/null 2>&1 || true # ignore failure.

/sbin/sysctl -q -e -w net.ipv6.conf.ens7.accept_ra=2
/sbin/sysctl -q -e -w net.ipv6.conf.ens7.autoconf=1
/bin/ip link set dev ens7 up
/lib/ifupdown/wait-for-ll6.sh
/sbin/dhclient -1 -6 -S -pf /run/dhclient6.ens7.pid -lf /var/lib/dhcp/dhclient6.ens7.leases -I -df /var/lib/dhcp/dhclient.ens7.leases ens7
/bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/000resolvconf
run-parts: executing /etc/network/if-up.d/ethtool
run-parts: executing /etc/network/if-up.d/ifenslave
+ [ inet6 = meta ]
+ [ ]
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/openssh-server
run-parts: executing /etc/network...

Read more...

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
tags: removed: sts-sponsor
tags: removed: patch
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ifupdown - 0.8.10ubuntu1.3

---------------
ifupdown (0.8.10ubuntu1.3) xenial; urgency=medium

  [ Wido den Hollander ]
  * Wait properly for Link-Local Address to go through DAD.
    (LP: #1447715)

 -- Dan Streetman <email address hidden> Tue, 17 Apr 2018 17:29:54 -0400

Changed in ifupdown (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for ifupdown has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.