Commissioning doesn't update the IP of PXE interface UI, maybe due to lease parser failure

[Andres Rodriguez]

Commissioning doesn't seem to be updating the IP of the PXE interface in the UI, and the error shown in the logs is a lease parser failure. See attached:

==> /var/log/maas/clusterd.log <==
2015-11-05 09:16:22-0500 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/provisioningserver/dhcp/leases.py", line 133, in check_lease_changes
            exc_info = sys.exc_info()
          File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
            self.gen.throw(type, value, traceback)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/shell.py", line 316, in objectfork
            yield pid, recv, send
          File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
            self.gen.throw(type, value, traceback)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/shell.py", line 222, in pipefork
            yield pid, fin, fout
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/shell.py", line 316, in objectfork
            yield pid, recv, send
          File "/usr/lib/python2.7/dist-packages/provisioningserver/dhcp/leases.py", line 124, in check_lease_changes
            send(parse_leases_file())
          File "/usr/lib/python2.7/dist-packages/provisioningserver/dhcp/leases.py", line 98, in parse_leases_file
            return fstat(leases_file.fileno()).st_mtime, parse_leases(contents)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/dhcp/leases_parser_fast.py", line 79, in parse_leases
            results.append((entry.host, entry.hardware.mac))
        exceptions.AttributeError: 'str' object has no attribute 'mac'

[Jeffrey C Jones (trapnine)]

Tried to replicate on trunk with virtual machines with single and multiple NIC's. No error and PXE interface IP looks good. Does this require a special setup?

[Matt Dirba (5qxm)]

I ran into a broken dhcp.leases file in maas 1.9 as well.

# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import provisioningserver
>>> parse_leases_file
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'parse_leases_file' is not defined
>>> from provisioningserver.dhcp.leases import parse_leases_file
>>> parse_leases_file()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "provisioningserver/dhcp/leases.py", line 98, in parse_leases_file
    return fstat(leases_file.fileno()).st_mtime, parse_leases(contents)
  File "provisioningserver/dhcp/leases_parser_fast.py", line 79, in parse_leases
    results.append((entry.host, entry.hardware.mac))
AttributeError: 'str' object has no attribute 'mac'
>>>

I was able to avoid the error by modifying /usr/lib/python2.7/dist-packages/provisioningserver/dhcp/leases_parser_fast.py as follows:
79c79,80
< if not has_expired(entry, now):
---
> if not has_expired(entry, now) and entry.hardware:
> # print('entry={}\nhardware{}\n'.format(entry.hardware, entry))

Unfortunately I did not save the dhcp.leases file. At the time of the failure I had servers failing to commission. Undoing the change now, I am unable to recreate the error.

[Andres Rodriguez (andreserl)]

@Matt,

Provided that this bug has a patch attached and is released with the latest version of MAAS 1.9 (1.9.2), I'm going to mark this Fix Released.

If you could upgrade to 1.9.2 and re-test, and if your problem still remains, could you please re-open it?

Thanks!

[LaMont Jones (lamont)]

@Matt,

Inside of "maas-region-admin dbshell", can you run the following query? (It should return zero rows..)

select username,sip.ip,sip.created,sip.updated from maasserver_staticipaddress as sip join auth_user on user_id=auth_user.id join maasserver_nodegroupinterface on management != 0 and inet(sip.ip) >= inet(ip_range_low) and inet(sip.ip) <= inet(ip_range_high) where alloc_type=4 order by sip.ip;

If it returns any rows, those user IP reservations should be released. (The bug is that we let you move the dynamic range on top of user reserved IPs.) If it returns zero rows, and you are still experiencing the issue, it's most interesting.

[LaMont Jones (lamont)]

If the above SELECT returns any rows, then this is a duplicate of bug 1536604.

[Matt Dirba (5qxm)]

@Andres Sounds good.

@LaMont The SQL query did not return anything.

@Anyone If a machine fails to commission or deploy does MAAS still release the IP address assigned to the host if the host is still holding on to the lease for the IP?