Assigning sticky IP address to node doesn't update controlers dhcp config

Bug #1423931 reported by Martin Nowack on 2015-02-20
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Blake Rouse
1.7
Critical
Blake Rouse

Bug Description

Reproduce issue:

1. Assign static ip addresses using

maas cluster node claim-sticky-ip-address node_id requested_ip=xxx.xxx.xxx.xxx

2. Runs successful as IP is shown in returned json

3. Acquiring node

Result:
Node gets dynamic IP assigned.

Expected:
Node gets static IP assigned

According to documentation, it's not explicitly stated that i need to add the node description to dhcp.template.
As a workaround, If I add an appropriate entry, it works.

Installed Packages
rc maas 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server all-in-one metapackage
ii maas-cli 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS command line API tool
ii maas-cluster-controller 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server cluster controller
ii maas-common 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server common files
ii maas-dhcp 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS DHCP server
ii maas-dns 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS DNS server
ii maas-proxy 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS Caching Proxy
rc maas-region-controller 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server complete region controller
ii maas-region-controller-min 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS Server minimum region controller
ii python-django-maas 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server Django web framework
ii python-maas-client 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS python API client
ii python-maas-provisioningserver 1.7.1+bzr3341-0ubuntu1~utopic1 all MAAS server provisioning libraries

Related branches

Blake Rouse (blake-rouse) wrote :

The static ip address configuration is not stored in the DHCP config. It is set using omshell from MAAS, that information is stored in the dhcp.leases file, which the dhcp server parses when it restarts. The leases file is the dhcpd database.

Changed in maas:
status: New → Invalid
Raphaël Badin (rvb) wrote :

> The static ip address configuration is not stored in the DHCP config. It is set using omshell from MAAS, that
> information is stored in the dhcp.leases file, which the dhcp server parses when it restarts. The leases file is the dhcpd database.

Still, the node should have gotten a static IP so we need to investigate.

Changed in maas:
status: Invalid → New
assignee: nobody → Mike Pontillo (mpontillo)
importance: Undecided → Critical
Mike Pontillo (mpontillo) wrote :

Thank you for the bug report. I'll see if I can confirm the issue.

I noticed you have the Utopic packages installed; I'm curious if you have tried Trusty?

Martin Nowack (martin-nowack) wrote :

Thanks. No, I didn't try Trusty - started with the Utopic.

Mike Pontillo (mpontillo) wrote :

Martin, I'm trying to create a test case for this that matches your environment. Can you provide the IP address ranges you used for the dynamic and static DHCP ranges in your cluster configuration, the expected (assigned static) IP address, and the actual (assigned by DHCP) values? (obfuscated IP addresses are okay, if this information is sensitive in your environment.)

It may also be helpful for me to see an example of the specific changes you made to dhcp.template (so we don't have to duplicate effort in case it contributes to a fix).

Finally, can you confirm that (under your cluster interface configuration) you have "DHCP" or "DHCP and DNS" selected? (it defaults to "Unmanaged", so I just wanted to double-check.)

Thanks in advance.

Mike Pontillo (mpontillo) wrote :

Confirmed on Utopic, but for me the behavior may have been different than described. A few issues I saw:

(1) 'requested_ip' is incorrect usage for this API. It should be 'requsted_address'. However, because the parameter is optional, when 'requested_ip' instead of 'requested_address', an IP address in the static range was assigned (but not the IP address that was requested on the command line).

(2) The static address was not written to the DHCP leases database, so the node retained its dynamic IP address.

(3) After claiming the IP address with the proper 'requested_address' syntax, the node retained the previous IP (dynamically allocated from the static pool) rather than switching to the one I requested.

Changed in maas:
status: New → Confirmed
Changed in maas:
status: Confirmed → Triaged
milestone: none → 1.7.2
Martin Nowack (martin-nowack) wrote :

You're right about requested_ip vs. requested_address. I just mistyped while filling out the bug report.
maas command line suggested the correct syntax.
Do you still need any other information?

Thanks a lot for your help.

Mike Pontillo (mpontillo) wrote :

Thanks Martin. I will continue to triage this issue on Utopic. If you have a chance, could you try MAAS on Trusty? I'd like to narrow the problem down between the MAAS code and the ISC DHCP server. (we communicate the lease directly to the DHCP server; perhaps there is a problem there on Utopic that does not occur on Trusty)

Mike Pontillo (mpontillo) wrote :

We should have seen a log in the following format if we fail to create a DHCP host mapping:

    Could not create host map for %s with address %s

In my testing, I'm not seeing that. So that means we either didn't try to create a DHCP mapping in the first place, or we tried and it silently failed.

I do see the following errors in /var/log/syslog on the DHCP server, but I'm not certain these correlate to when the host maps should be added.

Feb 20 12:49:11 maas-utopic dhcpd: Can't create new lease file: Permission denied
Feb 20 12:49:11 maas-utopic dhcpd: lease 192.168.21.101 end changed from 1424503739 to 0

I plan to add some additional logging and re-test.

SecurityFun23 (securityfun23) wrote :

I'm seeing the same (or at least similar) behavior on Trusty. When I run sudo maas my-maas node claim-sticky-ip-address system_id mac_address="XX:XX:XX:XX:XX:XX" requested_address="a.b.c.d", it only updates the maas, but not the dhcp, so I have to manually add items to the /etc/maas/templates/dhcp/dhcpd.conf.template so that the nodes boot off their static addresses rather than get a dynamic address. I put entries in like:

host serever1 {
     hardware ethernet XX:XX:XX:XX:XX:XX;
     fixed-address a.b.c.d;
}

Its unclear to me how MAAS is supposed to support these static addreses vs dynamic ...

Martin Nowack (martin-nowack) wrote :

I tried it on Trusty as well - the same issue there.

@securityfun23, if i understand it well, entries should *not* be placed explicitly in the dhcpd.conf file, instead they get automatically injected into the lease pool. This avoids updating that file all the time.

SecurityFun23 (securityfun23) wrote :

Martin,

Thanks! I did see something about the leases file in this thread, but I don't see anything in the MAAS python scripts about the claim-sticky-ip-address command. Perhaps there is an additional command that needs to be executed via omshell (as mention by Blake above) for this to take effect. However, if that's the case, I have not seen this documented anywhere ...

Martin Nowack (martin-nowack) wrote :

Found the issue:

Even a sticky IP address is assigned, it is not used while starting the machine and therefore the DHCP information is not updated (updated with an empty set).

The MP https://code.launchpad.net/~martin-nowack/maas/fix-sticky-ip fixes the problem for me.

I'm using a similar solution for the 1.7 branch.

Mike Pontillo (mpontillo) wrote :

Thanks for the patch, and for helping to narrow the issue.

According to the documentation for claim_static_ips(), the DHCP host maps are supposed to be updated by the caller after the IP address has been claimed. In this case, I think the DHCP maps should have been updated when the IP address was initially claimed using the CLI, but were not. (which is why your workaround helps, so it can re-issue the claim again, when the node starts.)

There's a good argument to be made for claiming IP addresses at the last possible moment (when you're starting the node): there's less time for a potential race condition to occur. (for example, if you're running the cluster service on another machine, and that machine crashes and restores a backup before the node gets a chance to start)

As you point out, the current code calls claim_static_ip_addresses(), but doesn't find any claims. Note that it only checks for AUTO addresses, and upon not finding any, fails to update the DHCP server. (But in the current design, the DHCP server should have been updated already.)

I'll make some comments on the pull request.

Changed in maas:
milestone: 1.7.2 → 1.7.3
Changed in maas:
assignee: Mike Pontillo (mpontillo) → Blake Rouse (blake-rouse)
status: Triaged → In Progress
milestone: 1.7.3 → next
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl) wrote :

This bug has been reported and fixed on upstream MAAS. However, provided that the bug was listed on the debian changelog, this appears as needing verification for pending SRU [1]. This bug did not affect current MAAS in Ubuntu, hence setting this to verification-done to unblock pending SRU.

[1]:http://people.canonical.com/~ubuntu-archive/pending-sru.html

Changed in maas:
status: Fix Committed → Fix Released
tags: added: verification-done
Changed in maas:
milestone: next → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers