Charm errs if its neighbor's IP does not fit in a local interface subnet

[Vern Hart]

I have a deployment where ceph-access-space and ceph-cluster-space both have two subnets.

For instance, ceph-access-space has both:

  172.31.250.0/23
  172.31.252.0/23

I've deployed an ceph-rbd-mirror charm on each subnet but the charm errors when scanning the ceph-access-space. I added debugging output in resolve_network_cidr that reports the ip_address and the netmask. You can see it in this stack trace with :: at the beginning. This is run on the node that has an IP in 172.31.250.0/23.

The workaround I used was to add "if netmask != None:" before the return statement of resolve_network_cidr (in .venv/lib/python3.6/site-packages/charmhelpers/contrib/network/ip.py). I guess that means this may be a charmhelpers bug.

inactive
Unit /<email address hidden> is masked, ignoring.
creating /var/lib/ceph/rbd-mirror/remote.client.rbd-mirror.juju-1f4aa0-8-lxd-0.keyring
added entity client.rbd-mirror.juju-1f4aa0-8-lxd-0 auth(key=AQA0d3dfA/ieCBAA5vJMWJ78hR112Zv9YCEinA==)
creating /var/lib/ceph/rbd-mirror/ceph.client.rbd-mirror.juju-1f4aa0-8-lxd-0.keyring
added entity client.rbd-mirror.juju-1f4aa0-8-lxd-0 auth(key=AQAiandf9fx+DxAAPj+ARRLsT4cwhlNwGlu/5g==)
:: 172.31.250.17 255.255.254.0
:: 172.31.250.17 255.255.254.0
:: 172.31.250.17 255.255.254.0
:: 172.31.252.22 None
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 803, in parse_ip_network
    prefixlen = int(val2)
ValueError: invalid literal for int() with base 10: 'None'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/strategy/ipv4.py", line 125, in str_to_int
    return _struct.unpack('>I', _inet_pton(AF_INET, addr))[0]
OSError: illegal IP address string passed to inet_pton

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 311, in __init__
    self._value = self._module.str_to_int(addr, flags)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/strategy/ipv4.py", line 129, in str_to_int
    raise AddrFormatError('%r is not a valid IPv4 address string!' % addr)
netaddr.core.AddrFormatError: 'None' is not a valid IPv4 address string!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 928, in __init__
    implicit_prefix, flags)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 810, in parse_ip_network
    mask = IPAddress(val2, module.version, flags=INET_PTON)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 314, in __init__
    % (addr, self._module.version))
netaddr.core.AddrFormatError: base address 'None' is not IPv4

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hooks/ceph-remote-relation-changed", line 22, in <module>
    main()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/charm/reactive/ceph_rbd_mirror_handlers.py", line 78, in render_stuff
    charm_instance.render_with_interfaces(args)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 834, in render_with_interfaces
    charm_instance=self))
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 812, in render_configs
    _render(os.path.basename(conf))
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 809, in _render
    perms=self.permission_override_map.get(conf) or 0o640,
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charmhelpers/core/templating.py", line 85, in render
    content = template.render(context)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "templates/remote.conf", line 16, in top-level template code
    public network = {{ ceph_remote.public_network }}
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/jinja2/environment.py", line 430, in getattr
    return getattr(obj, attribute)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms_openstack/adapters.py", line 162, in <lambda>
    self.relation, name)))(name))
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/charm/hooks/relations/ceph-rbd-mirror/requires.py", line 200, in public_network
    return ch_ip.resolve_network_cidr(public_addr)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charmhelpers/contrib/network/ip.py", line 234, in resolve_network_cidr
    return str(netaddr.IPNetwork("%s/%s" % (ip_address, netmask)).cidr)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 938, in __init__
    raise AddrFormatError('invalid IPNetwork %s' % addr)
netaddr.core.AddrFormatError: invalid IPNetwork 172.31.252.22/None

[Vern Hart (vern)]

--- ../.venv/lib/python3.6/site-packages/charmhelpers/contrib/network/ip.py.orig 2020-10-02 21:36:12.654080192 +0000
+++ ../.venv/lib/python3.6/site-packages/charmhelpers/contrib/network/ip.py 2020-10-02 21:36:24.110214789 +0000
@@ -230,7 +230,8 @@
     configured network interfaces
     '''
     netmask = get_netmask_for_address(ip_address)
- return str(netaddr.IPNetwork("%s/%s" % (ip_address, netmask)).cidr)
+ if netmask != None:
+ return str(netaddr.IPNetwork("%s/%s" % (ip_address, netmask)).cidr)

 def format_ipv6_addr(address):

[Vern Hart (vern)]

Subscribing field-medium as we have a local-charm workaround.

[Billy Olsen (billy-olsen)]

Fixing this in the charm-interface-ceph-rbd-mirror location to handle the AddrFormatError. Its possible that this will ripple to lots of other charms as well and it should be contained to the specific place for the time being.

[Billy Olsen (billy-olsen)]

Committed in this review:

https://review.opendev.org/c/openstack/charm-interface-ceph-rbd-mirror/+/784456