Default flag change in netaddr breaks (at least) unit tests

Bug #2054203 reported by Takashi Kajinami
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
In Progress
High
Dr. Jens Harbott

Bug Description

netaddr 1.0.0 changed the default parsing mode from INET_ATON to INET_PTON[1]. This was initially added to netaddr.IPAddress and then later was applied to netaddr.IPNetwork in 1.1.0 [2]

While we attempted to bump netaddr to 1.0.1, we noticed this change broke some of the unit tests in neutron.

https://zuul.opendev.org/t/openstack/build/8cfad48dcfb84be893fe78a1f965c5e6

(example)
```
neutron.tests.unit.agent.l3.extensions.test_ndp_proxy.NDPProxyExtensionDVRTestCase.test__get_snat_idx_ipv4
----------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/ip/__init__.py", line 346, in __init__
    self._value = self._module.str_to_int(addr, flags)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/strategy/ipv4.py", line 120, in str_to_int
    raise error
    netaddr.core.AddrFormatError: '101.12.13.00' is not a valid IPv4 address string!

During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/ip/__init__.py", line 1019, in __init__
    value, prefixlen = parse_ip_network(module, addr, flags)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/ip/__init__.py", line 899, in parse_ip_network
    ip = IPAddress(val1, module.version, flags=INET_PTON)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/ip/__init__.py", line 348, in __init__
    raise AddrFormatError(
    netaddr.core.AddrFormatError: base address '101.12.13.00' is not IPv4

During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/l3/test_dvr_local_router.py", line 549, in test__get_snat_idx_ipv4
    snat_idx = ri._get_snat_idx(ip_cidr)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_local_router.py", line 412, in _get_snat_idx
    net = netaddr.IPNetwork(ip_cidr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py311/lib/python3.11/site-packages/netaddr/ip/__init__.py", line 1028, in __init__
    raise AddrFormatError('invalid IPNetwork %s' % (addr,))
    netaddr.core.AddrFormatError: invalid IPNetwork 101.12.13.00/24
```

An "easy" fix is adding flags=INET_ATON to all places. (note that INET_ATON was added in netaddr 0.10.0) but I'd like to ask someone from the neutron team to look into this and check whether we really have to use INET_ATON or fix unit test side to apply the more strict rule.

[1] https://netaddr.readthedocs.io/en/latest/changes.html#release-1-0-0
[2] https://netaddr.readthedocs.io/en/latest/changes.html#release-1-1-0

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :
Changed in neutron:
assignee: nobody → Dr. Jens Harbott (j-harbott)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

One thing to discuss is whether the API allowing "101.12.13.00/24" was actually a bug or whether it needs to be considered an (undocumented?) feature and changing to the stricter verification would be a regression.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Added to the "on-demand" section of tomorrow's meeting: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda

Revision history for this message
Takashi Kajinami (kajinamit) wrote :
Download full text (4.0 KiB)

Please note this may affect wider patterns which were previously accepted.

For example with netaddr < 1.0.0, "10/8" is a valid representation of subnet and is translated to 10.0.0.0/8 .

Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
>>> netaddr.IPNetwork('10/8')
IPNetwork('10.0.0.0/8')

However with netaddr >= 1.0.0, which uses INET_PTON, this is no longer accepted

>>> import netaddr
>>> netaddr.IPNetwork('10/8')
Traceback (most recent call last):
  File "/home/tkajinam/venv/lib/python3.11/site-packages/netaddr/strategy/ipv4.py", line 124, in str_to_int
    return _struct.unpack('>I', _inet_pton(AF_INET, addr))[0]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: illegal IP address string passed to inet_pton

Currently with netaddr < 1.0.0, neutron supports that INET_ATON compliant format.

```
[root@localhost ~]# openstack subnet create --debug --network testnetwork --subnet-range 10/8 testsubnet
...
REQ: curl -g -i -X POST http://127.0.0.1:9696/v2.0/subnets -H "Content-Type: application/json" -H "User-Agent: openstacksdk/2.1.0 keystoneauth1/5.5.0 python-requests/2.25.1 CPython/3.9.18" -H "X-Auth-Token: {SHA256}f5f8310eff9da291b1ed3a851365033bc771153d5f148ecd59b969d2d32ce32a" -d '{"subnet": {"name": "testsubnet", "network_id": "156e8829-3b7c-45b1-8678-c35d625e0af4", "ip_version": 4, "cidr": "10/8"}}'
http://127.0.0.1:9696 "POST /v2.0/subnets HTTP/1.1" 201 608
RESP: [201] Connection: keep-alive Content-Length: 608 Content-Type: application/json Date: Tue, 20 Feb 2024 15:10:06 GMT X-Openstack-Request-Id: req-455d7d7b-8de3-45ab-b75c-a6b9f505838d
RESP BODY: {"subnet":{"id":"3ba2fa27-5931-4acc-bba5-3383232cd112","name":"testsubnet","tenant_id":"2e1290f8bdc44d35bce981db68fb54a2","network_id":"156e8829-3b7c-45b1-8678-c35d625e0af4","ip_version":4,"subnetpool_id":null,"enable_dhcp":true,"ipv6_ra_mode":null,"ipv6_address_mode":null,"gateway_ip":"10.0.0.1","cidr":"10.0.0.0/8","allocation_pools":[{"start":"10.0.0.2","end":"10.255.255.254"}],"host_routes":[],"dns_nameservers":[],"description":"","service_types":[],"tags":[],"created_at":"2024-02-20T15:10:05Z","updated_at":"2024-02-20T15:10:05Z","revision_number":0,"project_id":"2e1290f8bdc44d35bce981db68fb54a2"}}
POST call to network for http://127.0.0.1:9696/v2.0/subnets used request id req-455d7d7b-8de3-45ab-b75c-a6b9f505838d
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 10.0.0.2-10.255.255.254 |
| cidr | 10.0.0.0/8 |
| created_at | 2024-02-20T15:10:05Z |
| description | |
| dns_nameservers | |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.0.0.1 |
| host_routes | |
| id | 3ba2fa27-5931-4acc-bba5-3383232cd112...

Read more...

Revision history for this message
Takashi Kajinami (kajinamit) wrote (last edit ):

While we can recover the previous behavior by setting flags arg for IPAddress, I noticed there is no equivalent mechanism for IPNetwork ...

EDIT: I proposed https://github.com/netaddr/netaddr/pull/377

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/909317
Committed: https://opendev.org/openstack/neutron/commit/2ed9834047897d6b13a09ffa9cbeed2a4c0cc65b
Submitter: "Zuul (22348)"
Branch: master

commit 2ed9834047897d6b13a09ffa9cbeed2a4c0cc65b
Author: Dr. Jens Harbott <email address hidden>
Date: Sun Feb 18 12:42:40 2024 +0100

    Fix invalid IP address representation in unit test

    An IP address may not have a leading zero in any of its octets, this is
    getting enforced by the latest netaddr library.

    Partial-Bug: 2054203
    Change-Id: I15cd049de1511a9b52e8e28bccec87060c2f1411

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron-lib (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/910331

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.