1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases databases

Bug #1069570 reported by Thiago Martins on 2012-10-21
50
This bug affects 7 people
Affects Status Importance Assigned to Milestone
MAAS
Critical
Unassigned
1.3
Critical
Julian Edwards
juju-core
Undecided
Unassigned
isc-dhcp (Ubuntu)
Undecided
Unassigned
Declined for Quantal by Stéphane Graber
Precise
Undecided
Unassigned
Raring
Undecided
Unassigned
Saucy
Undecided
Unassigned
maas (Ubuntu)
High
Adam Stokes
Declined for Quantal by Stéphane Graber
Precise
Medium
Unassigned
Raring
Undecided
Unassigned
Saucy
High
Adam Stokes

Bug Description

[Impact]
 During my new tests, I tried to create two nodes simultaneously (both via DHCP + PXE, not by clicking "+Add node") and I hit this problem:

 Same MAC Address, two different IPs:

# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.4

server-duid "\000\001\000\001\030\023\243\007RT\000\360\236\231";

lease 192.168.50.1 {
  starts 5 2012/10/19 06:31:21;
  ends 5 2012/10/19 18:31:21;
  cltt 5 2012/10/19 06:31:21;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 52:54:00:cd:6b:a2;
  uid "\001RT\000\315k\242";
}
lease 192.168.50.2 {
  starts 5 2012/10/19 06:31:43;
  ends 5 2012/10/19 18:31:43;
  cltt 5 2012/10/19 06:31:43;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 52:54:00:cd:6b:a2;
  client-hostname "maas-enlist";
}
lease 192.168.50.2 {
  starts 5 2012/10/19 06:31:43;
  ends 5 2012/10/19 18:31:43;
  cltt 5 2012/10/19 06:31:43;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 52:54:00:cd:6b:a2;
  client-hostname "maas-enlist";
}
host 192.168.50.1 {
  dynamic;
  hardware ethernet 52:54:00:cd:6b:a2;
  fixed-address 192.168.50.1;
}
lease 192.168.50.1 {
  starts 5 2012/10/19 06:31:21;
  ends 5 2012/10/19 06:34:48;
  tstp 5 2012/10/19 06:34:48;
  cltt 5 2012/10/19 06:31:21;
  binding state free;
  hardware ethernet 52:54:00:cd:6b:a2;
  uid "\001RT\000\315k\242";
}

 1- DHCP gives address 192.168.50.1 to my machine on its first boot;

 2- After some time, I think that after some "reboots / elist / don'know for sure or even after commissioning" it; the IP have changed to 192.168.50.2 and got registered at DNS (192-168-50-2.test.com) that way;

 3- In the end of the day, after allocate it to root and start using it (the nodes), the DHCP gives again the IP 192.168.50.1 to my node, but DNS still remains with 192.158.50.2...

 ---

 My second node, first starts with 192.168.50.3, got registered with 192.168.50.4 at DNS (and at MaaS Web GUI) but the DHCP finish with 192.168.50.3...

[Test Case]
In a virtual environment:
- Add 2 nodes
- PXE boot the instances
- verify dhcp leases have 1 mac address for 2 different IPs

After applying latest debdiff alter the following:

   * modify /etc/maas/dhcpd.conf to include 'ignore-client-uids true'

re-test by adding 2 additional nodes to verify the mac address is unique between both nodes.

[Regression Potential]
Low, as it adds another optional arguement to disable checking client-uids

-- snip --

 I'll not try maas-dns again... Too many bugs... =(

 I think that MaaS needs some kind of "glue" between DHCP and DNS to keep it in sync.

Tks!
Thiago

Thiago Martins (martinx) on 2012-10-21
summary: - DNS is out of "sync" with DHCP leases databases
+ 1 MAC Address, two IPs - DNS is out of "sync" with DHCP leases
+ databases, I think...
Thiago Martins (martinx) on 2012-10-22
summary: - 1 MAC Address, two IPs - DNS is out of "sync" with DHCP leases
+ 1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases
databases, I think...

I think I found something related to this problem...

After start enlisting my second node, I saw this on my syslog:

---
Oct 22 00:58:34 seed1 dhcpd: DHCPDISCOVER from 52:54:00:7a:58:42 via eth0
Oct 22 00:58:35 seed1 dhcpd: DHCPOFFER on 192.168.50.2 to 52:54:00:7a:58:42 via eth0
Oct 22 00:58:36 seed1 dhcpd: Can't create new lease file: Permission denied <- ### PROBLEM!!! ###
Oct 22 00:58:36 seed1 dhcpd: DHCPREQUEST for 192.168.50.2 (192.168.63.250) from 52:54:00:7a:58:42 via eth0
Oct 22 00:58:36 seed1 dhcpd: DHCPACK on 192.168.50.2 to 52:54:00:7a:58:42 via eth0
---

After couple reboots (I think 1 reboot) (before allocating it):

---
Oct 22 00:59:05 seed1 dhcpd: DHCPDISCOVER from 52:54:00:7a:58:42 via eth0
Oct 22 00:59:06 seed1 dhcpd: DHCPOFFER on 192.168.50.3 to 52:54:00:7a:58:42 (maas-enlist) via eth0
Oct 22 00:59:06 seed1 dhcpd: DHCPREQUEST for 192.168.50.3 (192.168.63.250) from 52:54:00:7a:58:42 (maas-enlist) via eth0
Oct 22 00:59:06 seed1 dhcpd: DHCPACK on 192.168.50.3 to 52:54:00:7a:58:42 (maas-enlist) via eth0
Oct 22 00:59:06 seed1 dhcpd: DHCPREQUEST for 192.168.50.3 (192.168.63.250) from 52:54:00:7a:58:42 (maas-enlist) via eth0
Oct 22 00:59:06 seed1 dhcpd: DHCPACK on 192.168.50.3 to 52:54:00:7a:58:42 (maas-enlist) via eth0
---

My node receive a new IP...

Looking forward into this...

Best,
Thiago

Thiago Martins (martinx) wrote :

Hi!

Do you guys think that this is a good idea:

sudo adduser dhcpd maas
chmod 775 /var/lib/maas/dhcp

to avoid the error:

"Oct 22 00:58:36 seed1 dhcpd: Can't create new lease file: Permission denied" <- ### PROBLEM!!! ###

Where is the source of this problem?! It appear only one in my syslog... And I have no idea if it is gone...

Thanks!
Thiago

Thiago Martins (martinx) wrote :

Hi!

 Every time I start a new node via DHCP / PXE (no by adding it by clicking "+Add node"), I say this:

--
Oct 22 02:43:24 seed1 dhcpd: DHCPDISCOVER from 52:54:00:8c:59:8a via eth0
Oct 22 02:43:25 seed1 dhcpd: DHCPOFFER on 192.168.50.4 to 52:54:00:8c:59:8a via eth0

Oct 22 02:43:26 seed1 dhcpd: Can't create new lease file: Permission denied

Oct 22 02:43:26 seed1 dhcpd: DHCPREQUEST for 192.168.50.4 (192.168.63.250) from 52:54:00:8c:59:8a via eth0
Oct 22 02:43:26 seed1 dhcpd: DHCPACK on 192.168.50.4 to 52:54:00:8c:59:8a via eth0
Oct 22 02:43:49 seed1 dhcpd: DHCPDISCOVER from 52:54:00:8c:59:8a via eth0

NEW IP !!!

Oct 22 02:43:50 seed1 dhcpd: DHCPOFFER on 192.168.50.5 to 52:54:00:8c:59:8a (maas-enlist) via eth0
Oct 22 02:43:50 seed1 dhcpd: DHCPREQUEST for 192.168.50.5 (192.168.63.250) from 52:54:00:8c:59:8a (maas-enlist) via eth0
Oct 22 02:43:50 seed1 dhcpd: DHCPACK on 192.168.50.5 to 52:54:00:8c:59:8a (maas-enlist) via eth0
Oct 22 02:43:50 seed1 dhcpd: DHCPREQUEST for 192.168.50.5 (192.168.63.250) from 52:54:00:8c:59:8a (maas-enlist) via eth0
Oct 22 02:43:50 seed1 dhcpd: DHCPACK on 192.168.50.5 to 52:54:00:8c:59:8a (maas-enlist) via eth0
---

 Node is on "maas-inlisting-node" state...

Best,
Thiago

Changed in maas:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 12.10-stabilization
Raphaël Badin (rvb) wrote :

Actually, I think you're seeing two distinct problems here:

- The first one is that the default hostname picked up during enlistment is IP-based and thus it assumes that the IP lease will stay the same. If the lease changes, this change is not reflected in the DNS config because this is resolved by a fixed A record in the config file (and not a CNAME record which is only written when the node has a non-ip-based hostname) A workaround is to actually rename your node to something not IP-based.

- The other problem (dhcpd: Can't create new lease file: Permission denied) is something I've never seen and needs investigation. I don't think that changing the permissions of /var/lib/maas/dhcp will solve the problem. The DHCP server that MAAS manages is setup exactly the same way that the default dhcp is setup (the one which has its lease files in /var/lib/dhcp). The fact that you're seeing this problem only once leads me to think that this happens when the DHCP server rewrites the dhcp lease file.

Here is a extract from the DHCP server documentation:
"""
The lease file is rewritten occasionally to prevent excessive file growth. A temporary file is created containing the current lease file, the last file is renamed with a ~ suffix (e.g. /var/lib/dhcpd/dhcpd.leases~), then the temporary file is renamed to the configured lease file name.
"""
I suspect that the temporary file is written in /var/lib/dhcpd/ and that something prevents that file from being written. Maybe the current apparmor profile is too restrictive (http://paste.ubuntu.com/1297105/). Thiago, do you see apparmor-related errors in the logs around the time the dhcp error pops up?

Thiago Martins (martinx) wrote :

Raphael, how can I monitor the AppArmor log files?!

Using aa-logprof ?

I'm sorry, I don't know nothing about AppArmor yet...

Tks!

Raphaël Badin (rvb) wrote :

The apparmor error(s) would be in /var/log/syslog. For instance, if I manually change the apparmor maas dhcp.d profile so that the lease file can't be read I get this in /var/log/syslog : http://paste.ubuntu.com/1297221/

Thiago Martins (martinx) wrote :

Hi!

 After upgrading, I can't find that "permission denied" message again...

 But, the dhcpd still gives two IPs for 1 MAC (only for my second node), look:

Oct 23 11:15:02 seed1 dhcpd: DHCPDISCOVER from 52:54:00:79:f2:d0 via eth0
Oct 23 11:15:03 seed1 dhcpd: DHCPOFFER on 192.168.50.2 to 52:54:00:79:f2:d0 via eth0
Oct 23 11:15:04 seed1 dhcpd: DHCPREQUEST for 192.168.50.2 (192.168.63.250) from 52:54:00:79:f2:d0 via eth0
Oct 23 11:15:04 seed1 dhcpd: DHCPACK on 192.168.50.2 to 52:54:00:79:f2:d0 via eth0

NEW IP:

Oct 23 11:15:28 seed1 dhcpd: DHCPDISCOVER from 52:54:00:79:f2:d0 via eth0
Oct 23 11:15:29 seed1 dhcpd: DHCPOFFER on 192.168.50.3 to 52:54:00:79:f2:d0 (maas-enlist) via eth0
Oct 23 11:15:29 seed1 dhcpd: DHCPREQUEST for 192.168.50.3 (192.168.63.250) from 52:54:00:79:f2:d0 (maas-enlist) via eth0
Oct 23 11:15:29 seed1 dhcpd: DHCPACK on 192.168.50.3 to 52:54:00:79:f2:d0 (maas-enlist) via eth0
Oct 23 11:15:29 seed1 dhcpd: DHCPREQUEST for 192.168.50.3 (192.168.63.250) from 52:54:00:79:f2:d0 (maas-enlist) via eth0
Oct 23 11:15:29 seed1 dhcpd: DHCPACK on 192.168.50.3 to 52:54:00:79:f2:d0 (maas-enlist) via eth0

Best,
Thiago

Thiago Martins (martinx) wrote :

And now, before clicking "Accept and commission"...

IP back to 192.168.50.2...

Oct 23 11:18:32 seed1 dhcpd: uid lease 192.168.50.2 for client 52:54:00:79:f2:d0 is duplicate on 192.168.48.0/20
Oct 23 11:18:32 seed1 dhcpd: DHCPDISCOVER from 52:54:00:79:f2:d0 via eth0
Oct 23 11:18:32 seed1 dhcpd: DHCPOFFER on 192.168.50.2 to 52:54:00:79:f2:d0 via eth0
Oct 23 11:18:33 seed1 dhcpd: uid lease 192.168.50.2 for client 52:54:00:79:f2:d0 is duplicate on 192.168.48.0/20
Oct 23 11:18:33 seed1 dhcpd: DHCPDISCOVER from 52:54:00:79:f2:d0 via eth0
Oct 23 11:18:33 seed1 dhcpd: DHCPOFFER on 192.168.50.2 to 52:54:00:79:f2:d0 via eth0
Oct 23 11:18:35 seed1 dhcpd: Dynamic and static leases present for 192.168.50.2.
Oct 23 11:18:35 seed1 dhcpd: Remove host declaration 192.168.50.2 or remove 192.168.50.2
Oct 23 11:18:35 seed1 dhcpd: from the dynamic address pool for 192.168.48.0/20
Oct 23 11:18:35 seed1 dhcpd: uid lease 192.168.50.2 for client 52:54:00:79:f2:d0 is duplicate on 192.168.48.0/20
Oct 23 11:18:35 seed1 dhcpd: DHCPREQUEST for 192.168.50.2 (192.168.63.250) from 52:54:00:79:f2:d0 via eth0
Oct 23 11:18:35 seed1 dhcpd: DHCPACK on 192.168.50.2 to 52:54:00:79:f2:d0 via eth0

I'll accept it now to see what will happen...

BTW, the DNS was recorded with IP 192.168.50.3 (192-168-50-3.teste.com) !!!!!!!!!!! But the REAL IP of this node is 192.168.50.2.

Something is still wrong.

First node have IP 192.168.50.1 and DNS entry 192-168-50-1.teste.com.

Second node have IP 192.168.50.2 nd DNS entry 192-168-50-3.teste.com.

Cheers!
Thiago

Thiago Martins (martinx) wrote :

Hi!

 Instead of using this "Ready" node but, with wrong IP, I chose to disable the DNS Managed mode (I'll stay only DHCP for now)...

 And after deleting the nodes, I saw this:

==> /var/log/maas/celery.log <==
[2012-10-23 11:55:40,852: ERROR/MainProcess] Task provisioningserver.tasks.remove_dhcp_host_map[f36acf5d-76bf-4fa5-b3b3-ad18612d7fe8] raised exception: UnpickleableExceptionWrapper('subprocess', 'CalledProcessError', (), 'CalledProcessError()')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/execute/trace.py", line 181, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 301, in remove_dhcp_host_map
    omshell.remove(ip_address)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/omshell.py", line 192, in remove
    raise CalledProcessError(returncode, "omshell", output)
CalledProcessError: CalledProcessError()

Best,
Thiago

Thank you for the excellent feedback Thiago. I think we've got a problem with the permanent host mapping on your DHCPD, which is odd because I've not seen that happening before. I'll ask a few people for thoughts on this.

Thiago Martins (martinx) wrote :

AWESOME!!! ^_^
I'm here to try to make Ubuntu even better!! :-D
Also, let me know if you guys wants to access my test server to debug it, I can give root access to it without any problem.

Regards,
Thiago

Thiago Martins (martinx) wrote :

Guys,

 I don't know if this error message is related to this problem or not, so, I'm posting it here...

ERROR 2012-10-23 12:30:41,440 maas.maasserver ################################ Exception: File not found ################################
ERROR 2012-10-23 12:30:41,543 maas.maasserver Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/decorators/vary.py", line 19, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 167, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python2.7/dist-packages/piston/resource.py", line 165, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 294, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/api.py", line 928, in get_file
    raise MAASAPINotFound("File not found")
MAASAPINotFound: File not found

Tks,
Thiago

> MAASAPINotFound: File not found

This is caused when the media files (in MEDIA_ROOT) have been removed but the maas database entry is still referring to them. I presume you're seeing it when using juju?

Can you please paste your /etc/maas/dhcpd.conf file.

Also can you re-run the whole thing from scratch and while doing it run tcpdump:

sudo tcpdump -nieth0 -w/tmp/dump.pcap -s0 arp or icmp or port 67 or port 68

(assuming eth0 is the right interface)

Please paste the output from tcpdump and the resulting leases file.

We suspect that your pool of addresses in DHCPD is conflicting with some existing machines.

Thiago Martins (martinx) wrote :

My /etc/maas/dhcpd.conf file:

subnet 192.168.48.0 netmask 255.255.240.0 {
       next-server 192.168.63.250;
       filename "pxelinux.0";
       option subnet-mask 255.255.240.0;
       option broadcast-address 192.168.63.255;
       option domain-name-servers 192.168.63.250;
       option routers 192.168.63.254;
       range dynamic-bootp 192.168.50.1 192.168.50.250;
}
omapi-port 7911;
key omapi_key {
    algorithm HMAC-MD5;
    secret "qvwta915g1dUstfB5m4JpC9Gk2y1PJ07VYE5koWsXiIt2+QaLJFHtlQyopyS4Ze+ao5vqcUTruajzRb2NM+CNg==";
};
omapi-key omapi_key;

Thiago Martins (martinx) wrote :

Well, guys, I give up.
Hope next year (13.04) MaaS and Juju becomes STABLE.
Right now, it is useless. I just wipe out all the data I have. No more MaaS / Juju for me.

Regards,
Thiago

Sorry to hear that Thiago. Please come back when you are ready to try again.

Changed in maas:
status: Triaged → Incomplete
milestone: 12.10-stabilization → none
Robie Basak (racb) on 2012-10-25
Changed in maas (Ubuntu):
status: New → Incomplete
Thiago Martins (martinx) wrote :

Guys,

 Why this BUG was marked as incomplete?!

 It is very easy to reproduce it.

 Just install a brand new MaaS with DHCP and DNS management, start first node, start second node and BAM! Hit the BUG...

 I'll try again within 4~5 months to see how things are going...

Best!
Thiago

Hello Thiago

It is incomplete because we cannot recreate it. Hence we're waiting for you to provide the information requested in comment #15.

Launchpad Janitor (janitor) wrote :

[Expired for MAAS because there has been no activity for 60 days.]

Changed in maas:
status: Incomplete → Expired
Launchpad Janitor (janitor) wrote :

[Expired for maas (Ubuntu) because there has been no activity for 60 days.]

Changed in maas (Ubuntu):
status: Incomplete → Expired
Aram Hăvărneanu (aramh) wrote :

I've reopened this, it is affecting juju-core, it's very easy to reproduce (at least on virtual hardware), just see #20.

Changed in maas:
status: Expired → Confirmed
Robie Basak (racb) wrote :

#20 is exactly what happens in the MAAS QA lab every day. I don't believe that this bug has been seen there.

Regardless of that, I'd really like to see a tcpdump of all DHCP, ARP and ICMP traffic as requested in #15.

Anyone who is affected, please can you paste all the information requested in #15. Please also say which versions of every package you are using.

Robie - it could be a bug that's fixed in the code awaiting SRU. We are not testing older packages in the lab.

Changed in maas:
status: Confirmed → Incomplete
James Page (james-page) wrote :

I'm also seeing this in a virtual maas deployment we are testing; when the KVM instances boot the PXE boot gets one IP address lease and then the OS gets a second lease with a different IP address.

Result is that the dhpd leases file contains two ip addresses for each MAC address; order appears non-deterministic so whether the DNS entry for a given node is related to the PXE or OS lease is random.

James Page (james-page) on 2013-01-24
Changed in maas (Ubuntu):
status: Expired → Incomplete

This looks like it's BIOS-dependent then, which explains why some of us weren't able to recreate it.

Changed in maas:
status: Incomplete → Triaged
Thiago Martins (martinx) wrote :

"Nice" to see that this BUG affects more people! =P
I am completely out of time to test this even more... I'm doing my PoC with Openstack Folsom on Ubuntu 12.04...
Sorry to not tell you guys that on my test (#20), that I was testing MaaS under a KVM Virtual Machine...

Best,
Thiago

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu):
status: New → Confirmed
Robert Ayres (robert-ayres) wrote :

To update the ticket, I've reproduced using 12.10 KVM machines (have attached tcpdump and leases file as requested).

As described previously, iPXE gets a different IP address to the OS image. Seems iPXE sends a DHCP client-id parameter and the OS doesn't. The current version of isc-dhcp-server (4.2.4-1ubuntu10.1) treats each as a different request.

Unfortunately, I haven't found a way to stop iPXE sending a client-id parameter.
However, you can add the following PPA to your MAAS server:

https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages

which contains a patched DHCP server.
Updating to this, editing /etc/maas/dhcpd.conf to add the 'ignore-client-uids true;' to the subnet declaration, then restarting the MAAS DHCP server will fix the problem - client-ids will be ignored when assigning addresses.

Robert Ayres (robert-ayres) wrote :
Robert Ayres (robert-ayres) wrote :
Scott Moser (smoser) wrote :

Hi,
  rbasak asked me to update status here, so I'll try to do that.
  The branches linked to this bug contain a patch and fix that works for me. There are also packages in the ppa at https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages .

  I'd like to hear back from others as to if this solves their problem. To test this:
   * build the package or use the ppa to get the updated version of isc-dhcp
   * modify /etc/maas/dhcpd.conf to include 'ignore-client-uids true'

  To me, this seems like the correct solution, and a valid patch to isc-dchp. The only reason we haven't landed on this as "the solution", is that isc is a very un-available upstream, and bringing this patch into ubuntu means carrying (and supporting) quite likely without upstream support.

Dave Walker (davewalker) wrote :

To go further, upstream seem to be hostile to this in principle, each time the matter is raised - as it isn't RFC compliant. Inversely, dnsmasq has this option in core.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.2.4-5ubuntu2

---------------
isc-dhcp (4.2.4-5ubuntu2) raring; urgency=low

  * debian/patches/add-option-ignore-client-uids.patch:
    Add a new dhcpd.conf option 'ignore-client-uids'. (LP: #1069570)
 -- Scott Moser <email address hidden> Thu, 14 Mar 2013 17:49:10 -0400

Changed in isc-dhcp (Ubuntu):
status: Confirmed → Fix Released
description: updated
description: updated
Changed in maas (Ubuntu):
status: Incomplete → Invalid
Adam Stokes (adam-stokes) wrote :

This also contains a small change to 00List to fix previous changelog entry and not have its patch reverted.

Adam Stokes (adam-stokes) wrote :

This also contains a fix from the previous changelog to modify an apparmor profile. Patch was removed and applied directly to debian/apparmor-profile.dhcpd

Dave Chiluk (chiluk) wrote :

The above debdiff resolves the 00list issues for bug
https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1057358

Stéphane Graber (stgraber) wrote :

I'm declining this for precise and quantal SRU as the fix for this bug is the addition of a new configuration parameter and parameter number which is a feature change and therefore not suitable for SRU.

Dave Walker (davewalker) wrote :

@stgraber, whilst I am not happy with the latest debdiff.. I do think this issue requires closer consideration. This is the only way that has been determined to resolve multiple IP leases per boot (can consume up to 3 different ones, with some configurations). This means that pools are quickly exhausted, which is a real issue in larger deployments. Therefore, it would seem to be the best way to solve this bug.

Whilst this does introduce a new configuration option, it has a seemingly low impact - as it doesn't change defaults, nor require configuration changes to people upgrading - is a well documented change in the manpage. And importantly, solves a larger bug that has been an issue for some time.

Stéphane Graber (stgraber) wrote :

I'm just not convinced that there aren't ways to workaround the issue that's less invasive than SRUing this change.

Did you consider pattern-matching the client string from iPXE and setting a 30s lease for those? That should solve your pool exhaustion problem and is supported syntax from upstream dhcpd.

julian wang (zeratul-j) on 2013-05-16
information type: Public → Public Security
information type: Public Security → Public
Dave Walker (davewalker) wrote :

@stgraber, That is a good point. However, I'm not sure it's practical to bake a solution tied to using iPXE, even if chain loaded, still requires an initial DHCP lease to be consumed by the firmware PXE client.

I still maintain that a pretty good direction for this fix is the ignore-client-uid patch, which has additionally proved itself in Raring - which forces no changes on users, simply providing an option which can be opted into if needed.

However, If you have further suggestions on how this can be resolved in a generic manner it would be greatly appreciated.

Thanks

Nobuto Murata (nobuto) wrote :

@Scott Moser,

for the record, I had to tweak the package for precise on PPA[1] to try the workaround for this issue.

add-option-ignore-client-uids.dpatch was reverted during the build[2].
=====
reverting patch add-option-ignore-client-uids from ./ ... ok.
=====

I changed the patch order and did some refreshment forcibly[3], then certainly the workaround works for me.

[1] https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages
[2] https://launchpadlibrarian.net/138695896/buildlog_ubuntu-precise-amd64.isc-dhcp_4.1.ESV-R4-0ubuntu5.6%2Bnouid0_UPLOADING.txt.gz
[3] https://launchpad.net/~nobuto/+archive/build-test/+sourcepub/3230909/+listing-archive-extra

Scott Moser (smoser) wrote :

Nobuto,
  Thanks. Fixing.

Scott Moser (smoser) wrote :

Nobuto,
   Thanks again. I just fixed, and
 * refreshed against '4.1.ESV-R4-0ubuntu5.8' that is now in precise-updates.
 * pushed to lp:~smoser/ubuntu/precise/isc-dhcp/nouid
 * uploaded to https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages

Changed in maas:
status: Triaged → Fix Released
William Reade (fwereade) on 2013-06-14
Changed in juju-core:
status: New → Invalid
Changed in maas (Ubuntu):
status: Invalid → In Progress
assignee: nobody → Adam Stokes (adam-stokes)
importance: Undecided → Medium
importance: Medium → High
Changed in maas (Ubuntu):
status: In Progress → Invalid

Invalid?! :-P

On 31 July 2013 17:09, Adam Stokes <email address hidden> wrote:

> ** Changed in: maas (Ubuntu)
> Status: In Progress => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1069570
>
> Title:
> 1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases
> databases, I think...
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1069570/+subscriptions
>

This bug was fixed in the package maas - 1.4+bzr1539+dfsg-0ubuntu2

---------------
maas (1.4+bzr1539+dfsg-0ubuntu2) saucy; urgency=low

  * debian/patches/04-dhcp-lease-conflict.patch:
    Sets expiry lease time to 30 seconds (LP: #1069570)
 -- Adam Stokes <email address hidden> Wed, 31 Jul 2013 15:40:06 -0400

Changed in maas (Ubuntu):
status: Invalid → Fix Released
Adam Stokes (adam-stokes) wrote :

Hmm, so I marked this bug invalid b/c I created a new bug 1207102 with the same changes. This got uploaded anyway and referenced this bug.

Jonathan Davies (jpds) on 2013-09-11
summary: - 1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases
- databases, I think...
+ 1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases databases
Robbie Williamson (robbiew) wrote :

Apologies if this is a stupid question, but why was this declined for Saucy?

Scott Moser (smoser) wrote :

Robbie,
 I just fixed state. oddly I just noticed this too. Anyway, fix is in isc-dhcp in raring (in release pocket even) and any subsequent ubuntu (saucy). Fix for maas to utilize this is also in raring and saucy.

Changed in isc-dhcp (Ubuntu Raring):
status: New → Fix Released
Changed in maas (Ubuntu Raring):
status: New → Fix Released
Scott Moser (smoser) on 2013-09-11
Changed in maas (Ubuntu Raring):
status: Fix Released → Confirmed

Hello Thiago, or anyone else affected,

Accepted maas into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.2+bzr1373+dfsg-0ubuntu1~12.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in maas (Ubuntu Precise):
importance: Undecided → Medium
status: New → Triaged
status: Triaged → Fix Committed
tags: added: verification-needed
Matt Rae (mattrae) wrote :

I see for precise-proposed, the patch is to add a 30 second lease for pxe agents. This is resulting the lease during pxe being added to the leases file with a 30 second expiration as expected, but the expiration is not being honored.

We are seeing machines given the ip from the 30 second lease long after it had expired.

It appears dhcpd is not respecting the 30 second lease and still giving the wrong ip to nodes.

Example:

Below are the leases for a machine that came up with the wrong ip. The lease for 172.21.66.24 is supposed to be for 30 seconds, but it appears it was not being respected, since the node would come up with .24 after rebooting long after the lease should be expired.

host 172.21.66.24 {
  dynamic;
  hardware ethernet 52:54:00:e3:1d:47;
  fixed-address 172.21.66.24;
}
lease 172.21.66.24 {
  starts 4 2013/09/05 01:40:28;
  ends 4 2013/09/05 01:40:58;
  tstp 4 2013/09/05 01:40:58;
  cltt 4 2013/09/05 01:40:28;
  binding state free;
  hardware ethernet 52:54:00:e3:1d:47;
  uid "\001RT\000\343\035G";
}
lease 172.21.66.29 {
  starts 4 2013/09/05 01:40:40;
  ends 4 2013/09/05 13:40:40;
  tstp 4 2013/09/05 13:40:40;
  cltt 4 2013/09/05 01:40:40;
  binding state active;
  next binding state free;
  hardware ethernet 52:54:00:e3:1d:47;
  client-hostname "maas-enlist";
}

This is happening because MAAS added the host{} entry for its MAC before the lease expired. The 30 second thing is clearly a hack and doesn't work properly because it's a race condition with MAAS lease scanner.

The only real solution is to SRU the ignore-client-ids fix.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 1.2+bzr1373+dfsg-0ubuntu1~12.04.3

---------------
maas (1.2+bzr1373+dfsg-0ubuntu1~12.04.3) precise-proposed; urgency=low

  * MAAS Stable Release Update:
    - debian/patches/04-dhcp-lease-conflict.patch: Sets expiry lease time
      to 30 seconds (LP: #1069570)
 -- Andres Rodriguez <email address hidden> Thu, 05 Sep 2013 22:53:10 -0400

Changed in maas (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu Precise):
status: New → Confirmed
Rolf Leggewie (r0lf) wrote :

raring has seen the end of its life and is no longer receiving any updates. Marking the raring task for this ticket as "Won't Fix".

Changed in maas (Ubuntu Raring):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers