[2.1] netaddr raises IndexError when looking up some OUI registrations

[ptylenda]

Distro: Ubuntu 16.10
MAAS version: 2.1.2+bzr5555-0ubuntu1~16.10.1 500

For some node MAC addresses (I have a relatively old servers) I am unable to open MAAS dashboard page or Nodes page due to

"list index out of range"

error.

2017-01-09T15:39:21+0100 [maasserver.websockets.websockets#debug] Opening connection with IPv6Address(TCP, '::1', 53172)
2017-01-09T15:39:21+0100 [maasserver.websockets.protocol#critical] Error on request (18) machine.list: list index out of range

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 824, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 225, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 182, in list
    return super(MachineHandler, self).list(params)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 358, in list
    for obj in objs
  File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 358, in <listcomp>
    for obj in objs
  File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 205, in full_dehydrate
    return self.dehydrate(obj, data, for_list=for_list)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 187, in dehydrate
    obj, data, for_list=for_list)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/node.py", line 118, in dehydrate
    data["pxe_mac_vendor"] = obj.get_pxe_mac_vendor()
  File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 3021, in get_pxe_mac_vendor
    return get_vendor_for_mac(boot_interface.mac_address.get_raw())
  File "/usr/lib/python3/dist-packages/maasserver/utils/mac.py", line 20, in get_vendor_for_mac
    return data.oui.registration().org
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 478, in oui
    return OUI(self.value >> 24)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 98, in __init__
    self._parse_data(data, offset, size)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 145, in _parse_data
    record['org'] = line.split(None, 2)[2]
builtins.IndexError: list index out of range

For now I have worked around the issue by catching IndexError in "/usr/lib/python3/dist-packages/maasserver/utils/mac.py". After this workaround the Nodes page started working but Dashboard page required similar fix in "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py" in line 923

But this probably needs some investigation and fix - without the workaround I cannot use MAAS.

[LaMont Jones (lamont)]

Can you provide the first 3 octets of one of the bad MACs? If you're not sure which one, the following is sure to catch it:

sudo maas-region dbshell
select distinct substring(cast(mac_address as varchar) for 8) from maasserver_interface;

This would appear to be a bug in the python3-netaddr, which MAAS should probably armor itself against.

[LaMont Jones (lamont)]

I found some that exhibit the issue (00-22-59, 00-23-89) for starters.

[LaMont Jones (lamont)]

MAAS was incomplete in the workaround for Bug#1628761.

[Mike Pontillo (mpontillo)]

Hm. I think the root cause of this might be that the index is not updated often enough (that is, it could be a Debian package mismatch).

The `netaddr` package defines the following:

ls -la /usr/lib/python3/dist-packages/netaddr/eui/oui.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python3/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt

readlink -f /usr/lib/python3/dist-packages/netaddr/eui/oui.txt
/usr/share/ieee-data/oui.txt

The `ieee-data` provides the actual data:

dpkg -S /usr/share/ieee-data/oui.txt
ieee-data: /usr/share/ieee-data/oui.txt

... so if one package is updated but not the other, the index in `netaddr` will become incorrect. I ran some tests with `netaddr` upstream and it seems like the library works fine with any OUI, even OUIs with incorrectly encoded characters in them, etc.

To test this theory, you could run netaddr as follows to regenerate the indexes:

    sudo python3 -m netaddr.eui.ieee

You would need to restart MAAS after doing this, since netaddr keeps the indexes cached in memory.

[LaMont Jones (lamont)]

Interesting point. Here is what I see:

[xenial]
ii ieee-data 20150531.1 all OUI and IAB listings
ii python3-netaddr 0.7.18-1 all manipulation of various common netwo
% python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
'Guangzhou New Postcom Equipment Co.,Ltd.'

[zesty]
ii ieee-data 20160613.1 all OUI and IAB listings
ii python3-netaddr 0.7.18-2 all manipulation of various common network
% python3
Python 3.5.2+ (default, Dec 13 2016, 14:16:35)
[GCC 6.2.1 20161212] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 478, in oui
    return OUI(self.value >> 24)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 98, in __init__
    self._parse_data(data, offset, size)
  File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 145, in _parse_data
    record['org'] = line.split(None, 2)[2]
IndexError: list index out of range
>>>

That is netaddr is not sufficiently hardened against the newer ieee-data. While it's conceivable that the actual bug is in ieee-data, it would seem that hardening the code to better deal with it would be good.

The entry that breaks reads:
00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CN
while the same entry in the working file is:
  00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
  002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CHINA
(Note the leading pair of spaces...)

[Mike Pontillo (mpontillo)]

Thanks for the test case. Here's what I saw after running `sudo python3 -m netaddr.eui.ieee` on my system:

>>> import netaddr
>>> netaddr.EUI('00:22:59:11:22:33').oui.registration().org
'Guangzhou New Postcom Equipment Co.,Ltd.'

So I conclude that the indexes are out of date.

My guess is `netaddr` expects to ship with the entire OUI database, but the Debian maintainer didn't want several different copies of the OUI database around the filesystem.

[ptylenda (tylenda-piotr)]

Thanks for quick response, I see you already have the MACs which cause problems, if needed I can print the ones which caused problem at my side

[LaMont Jones (lamont)]

The actual source of the issue seems to be that netaddr indexes a file that it doesn't own (from ieee-data package), and then assumes that it won't change underneath it (either because of an ieee-data update, or a netaddr update in the case of a running process), which results in various exceptions and other errors from the code. I believe that a fresh bug is being filed against python-netaddr on the issue.

[Mike Pontillo (mpontillo)]

I filed two additional bugs to track this.

(1) For the netaddr package, so that the indexes remain consistent with the IEEE data:

https://launchpad.net/bugs/1655447

(2) For the bug in netaddr which allows a long-running process to retain an old cached index:

https://bugs.launchpad.net/netaddr/+bug/1655452

... also filed upstream, at:

https://github.com/drkjam/netaddr/issues/146