[2.1] netaddr assumes MAC OUI is ascii

Bug #1628761 reported by Mike Pontillo on 2016-09-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
High
Mike Pontillo
python-netaddr (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Zesty
Medium
Unassigned
Artful
Medium
Unassigned
Bionic
Medium
Unassigned

Bug Description

This causes a traceback when we try to get the MAC organization, as follows:

    --- <exception caught here> ---
      File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
        result = inContext.theWork()
      File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
        inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
      File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 827, in callInContext
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction
        return func_outside_txn(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier
        return func(*args, **kwargs)
      File "/usr/lib/python3.5/contextlib.py", line 30, in inner
        return func(*args, **kwds)
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 356, in list
        for obj in objs
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 356, in <listcomp>
        for obj in objs
      File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 205, in full_dehydrate
        return self.dehydrate(obj, data, for_list=for_list)
      File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/discovery.py", line 38, in dehydrate
        data["mac_organization"] = obj.mac_organization
      File "/usr/lib/python3/dist-packages/maasserver/models/discovery.py", line 228, in mac_organization
        return get_mac_organization(str(self.mac_address))
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 817, in get_mac_organization
        return get_eui_organization(EUI(mac))
      File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 802, in get_eui_organization
        registration = eui.oui.registration()
      File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 478, in oui
        return OUI(self.value >> 24)
      File "/usr/lib/python3/dist-packages/netaddr/eui/__init__.py", line 97, in __init__
        data = fh.read(size)
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    builtins.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2420: ordinal not in range(128)

Related branches

Mike Pontillo (mpontillo) wrote :

This seems to be a bug in netaddr and/or a side effect of corrupt entries, in oui.txt, depending on how you approach the problem.

In '/usr/lib/python3/dist-packages/netaddr/eui/oui.txt' there are entries such as:

  00-22-59 (hex) Guangzhou New Postcom Equipment Co.,Ltd.
  002259 (base 16) Guangzhou New Postcom Equipment Co.,Ltd.
                                No.3¡¡Guangpuxi Road,Guangzhou Science City,
                                Guangzhou Guangdong 510663
                                CHINA

Note the non-ASCII characters. (They don't look correct, so I doubt the entries in this file were imported with a consistent source encoding.) There are numerous examples of characters like this if you search through the file for characters with an 8th bit set.

summary: - netaddr assumes MAC OUI is ascii
+ [2.1] netaddr assumes MAC OUI is ascii
Changed in maas:
status: New → Triaged
importance: Undecided → High
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
milestone: none → 2.1.0
Gavin Panella (allenap) wrote :

FWIW, http://paste.ubuntu.com/23275209/ (a short Python 3 script) makes me fairly sure that everything in netaddr's oui.txt is UTF-8. The script tries to guess the encoding of each entry using chardet, ignores those guessed as ASCII or UTF-8, then prints each entry after being decoded using the guessed encoding and again using UTF-8. The UTF-8 result always appears to be the better choice: http://paste.ubuntu.com/23275206/.

I mention this because it means that an upstream fix is probably easy, and because we may be able to monkeypatch something in the meantime. I do think the defensive approach you've taken in lp:~mpontillo/maas/netaddr-assumes-mac-oui-ascii--bug-1628761 is the right one for now though.

Of course, the example in the bug description is an exception to the rule: I can't find anything to decode that and produce something reasonable, though UTF-8 does at least offer a non-crashing option. I suspect it's been mangled, maybe by copy-and-paste at some indeterminate time in the past, and the original is now as good as unrecoverable from oui.txt. Similar corruption appears in the oui.txt that the ieee-data package installs.

Changed in maas:
status: Triaged → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
LaMont Jones (lamont) wrote :

Likewise, as of 0.7.18-2, the mac prefixes 00:22:59, 00:23:89, and 5c:15:15 generate IndexError, because unicode is treated as ascii.

James Page (james-page) wrote :

Only impacts py3; if you force the open of the file to use UTF-8 encoding everything is OK.

Changed in python-netaddr (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
James Page (james-page) wrote :
Changed in python-netaddr (Ubuntu):
status: Confirmed → Triaged
James Page (james-page) wrote :

That's included in the next patch release .19 (we're on .18), which we'll get as soon as bionic syncs start happening.

James Page (james-page) wrote :

$ rmadison -u debian python-netaddr
python-netaddr | 0.7.7-1 | oldoldstable | source, all
python-netaddr | 0.7.12-2 | oldstable | source, all
python-netaddr | 0.7.12-2 | oldstable-kfreebsd | source, all
python-netaddr | 0.7.18-1~bpo8+1 | jessie-backports | source, all
python-netaddr | 0.7.18-2 | stable | source, all
python-netaddr | 0.7.18-2 | testing | source, all
python-netaddr | 0.7.19-1 | unstable | source, all

Changed in python-netaddr (Ubuntu Bionic):
status: Triaged → Fix Released
Changed in python-netaddr (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Medium
Changed in python-netaddr (Ubuntu Zesty):
status: New → Triaged
Changed in python-netaddr (Ubuntu Artful):
status: New → Triaged
Changed in python-netaddr (Ubuntu Zesty):
importance: Undecided → Medium
Changed in python-netaddr (Ubuntu Artful):
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers