Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\xf5\x88\x06', 98, 'Address already in use')

[Andres Rodriguez]

I'm seeing this error in rackd.log. There's no specific way to reproduce this error, but for context, I was simply running the CI at which point this error just appeared after all machines were deployed.

2018-07-27 00:41:46 provisioningserver.utils.services: [info] Starting beaconing for interfaces: {'virbr0'}
2018-07-27 00:41:46 provisioningserver.utils.services: [info] Beaconing process for virbr0 started.
2018-07-27 00:41:46 twisted.internet.defer: [critical] Unhandled error in Deferred:
2018-07-27 00:41:46 twisted.internet.defer: [critical]
 Traceback (most recent call last):
 Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\xf5\x88\x06', 98, 'Address already in use')

2018-07-27 00:41:46 provisioningserver.utils.services: [info] observe-beacons[virbr0]: tcpdump: listening on virbr0, link-type EN10MB (Ethernet), capture size 16384 bytes
2018-07-27 00:41:52 provisioningserver.utils.services: [info] New interface monitoring state: {'ens3': {'neighbour': False, 'mdns': True}, 'ens4': {'neighbour': True, 'mdns': True}, 'virbr0': {'neighbour': True, 'mdns': True}}
2018-07-27 00:41:52 provisioningserver.utils.services: [info] Starting neighbour discovery for interfaces: {'virbr0'}
2018-07-27 00:41:52 provisioningserver.utils.services: [info] Neighbour observation process for virbr0 started.
 2018-07-27 00:41:53 provisioningserver.utils.services: [info] observe-arp[virbr0]: tcpdump: listening on virbr0, link-type EN10MB (Ethernet), capture size 64 bytes

[Mike Pontillo (mpontillo)]

I believe this bug was caused because MAAS was picking up interfaces attached to bridges for the purpose of attaching VMs or containers, in addition to the bridges themselves.

That is, this should have been resolved with the following commit:

commit 842646142d42d95453f4b64c209f93b09f8aea7c
Author: Mike Pontillo <email address hidden>
Date: Fri Aug 24 21:08:12 2018 +0000

    LP: #1788780 - Ignore tunnel interfaces (such as those created by hypervisors) when gathering interfaces for MAAS management.

[Mike Pontillo (mpontillo)]

I'm going to set this to Incomplete unless it can be reproduced with the latest alpha release. Feel free to reopen it if you see it again, and I'll take another look.

[Roman Dolejší (rdolejsi)]

The same issue was reproduced on MAAS 2.9.0beta5 (visible in rackd.log) running in a virtual as a standard package (not snap):

2020-10-20 18:16:32 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: mgmt0, docker0, bond0, cni0.
2020-10-20 18:17:12 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.
2020-10-20 18:21:32 twisted.internet.defer: [critical] Unhandled error in Deferred:
2020-10-20 18:21:32 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\xc0\xa8\x14\x05', 98, 'Address already in use')

I don't believe this issue should be marked as duplicate of the bug #1788780 or its resolution should be revisited.

[Roman Dolejší (rdolejsi)]

The issue is intermittent, restarting maas-rackd "overcame" the issue (for now).

[to-hiro (to-hiro)]

Hello,

It seems having same issue.

2021-02-01 06:34:23 twisted.internet.defer: [critical] Unhandled error in Deferred:
2021-02-01 06:34:23 twisted.internet.defer: [critical]

Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n[*\x15', 98, 'Address already in use')
2021-02-01 06:34:39 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: (b'UNHANDLED', 'Unknown Error [DUB1OSSKMLA003:pid=13615:cmd=UpdateInterfaces:ask=7e205]'); interfaces: {'broam': {'index': 10, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': False, 'links': [{'mode': 'static', 'gateway': '10.91.42.1', 'address': '10.91.42.21/23'}], 'parents': ['bond0'], 'type': 'bridge', 'source': 'ipaddr'}, 'enp88s0f1': {'index': 7, 'mac_address': '68:05:ca:57:7d:f5', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp47s0f1': {'index': 5, 'mac_address': '68:05:ca:57:7d:b9', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'eno1': {'index': 2, 'mac_address': '7c:d3:0a:e5:36:c8', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp88s0f0': {'index': 6, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp0s20f0u1u6': {'index': 8, 'mac_address': '7e:d3:0a:e5:36:cf', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'lxdbr0': {'index': 11, 'mac_address': '16:0f:d3:ae:76:3d', 'enabled': True, 'monitored': True, 'links': [], 'parents': [], 'type': 'bridge', 'source': 'ipaddr'}, 'bond0': {'index': 9, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': True, 'links': [], 'parents': ['enp88s0f0', 'enp47s0f0'], 'type': 'bond', 'source': 'ipaddr'}, 'enp47s0f0': {'index': 4, 'mac_address': '68:05:ca:57:7d:b8', 'enabled': True, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'eno2': {'index': 3, 'mac_address': '7c:d3:0a:e5:36:c9', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}}

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1000, in updateInterfaces
    yield self._updateInterfaces(interfaces)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py"...

[Linus Walther (linwalth)]

We are having this problem with MAAS 3.3.3 (Using 3 RackDs and 3 RegionDs, they are not running stable, as are flapping in and out of functional services.

Our RackDs have 2 interfaces each. One for the untagged net and one for the OOB-Net to reach the BMCs.

2023-05-22 08:45:38 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-05-22 08:45:38 twisted.internet.defer: [critical]
 Traceback (most recent call last):
 Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x04\x00\x0c', 98, 'Address already in use')

Could this be a regression, potentially?
Besides, I do not think that this bug is a duplicate of https://bugs.launchpad.net/maas/+bug/1788780 as we are not using libvirt on our RackDs and still encounter the problem.

[Linus Walther (linwalth)]

Since this bug affects a different version than ours (3.3.3) I will create a new bugreport and link to this as duplicate. I am sorry for the changes to the ticket, as I am still just learning how to use launchpad :)

[Jerzy Husakowski (jhusakowski)]

Issue LP:2020798 seems related but occurs on a newer version of MAAS and has a different failure mode. This issue shows "Failed to update and/or record network interface configuration" which is not reproducible on the current MAAS.

At the least, we will improve the error message to make the root cause easier to diagnose, should this occur again.

[Heather Lemon (hypothetical-lemon)]

Linking another launchpad bug with similar errors being thrown https://bugs.launchpad.net/maas/+bug/2030781

[Muhammad Ahmad (ahmadfsbd)]

We are hitting a similar error on a customer environment after upgrading from Maas 2.9 to 3.2.

2023-09-12 06:55:19 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: brappformix, brext, broam.
2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'666\t', 98, 'Address already in use')

2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x0f\xa1\x05', 98, 'Address already in use')

2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x0f\xa8\n', 98, 'Address already in use')

2023-09-12 06:48:26 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: Expecting value: line 1 column 1 (char 0); interfaces: {'bond0': {'type': 'bond', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['eno12409np1', 'eno12399np0'], 'source': 'machine-resources', 'monitored': True}, 'bond0.402': {'type': 'vlan', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'vid': 402, 'monitored': False}, 'bond0.403': {'type': 'vlan', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'vid': 403, 'monitored': False}, 'brappformix': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted'}], 'enabled': True, 'parents': ['bond0.403'], 'source': 'machine-resources', 'monitored': False}, 'brext': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted'}], 'enabled': True, 'parents': ['bond0.402'], 'source': 'machine-resources', 'monitored': False}, 'broam': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted', 'gateway': 'Redacted'}, {'mode': 'static', 'address': 'Redacted', 'gateway': 'Redacted'}], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'monitored': False}, 'eno12399np0': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno12409np1': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno8303': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno8403': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resourc...

[Bartłomiej Poniecki-Klotz (barteus)]

We are affected by this bug when upgrading from MAAS 3.1 to 3.2 or 3.3

The workaround for us is to use 3.1/stable

[Jerzy Husakowski (jhusakowski)]

Muhammad & Bartlomiej, could you file a separate issue for the problem you're experiencing? Could you indicate what breaks or stops working for you?

The MulticastJoinError "Address already in use" trace may be unrelated or a red herring. With this issue, we will improve the tracing as it is confusing, but in previous occurrences of this trace MAAS continued to work. If you experience otherwise, that's a separate issue.

[maasuser1 (maasuser1)]

This problem also affects MAAS 3.4/Stable (3.4.0-14321-g.1027c7664).

```
2024-02-12 12:26:44 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-02-12 12:26:44 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        --- <exception caught here> ---
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
            error.trap(RemoteAmpError)
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
        twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=320093:cmd=ReportNeighbours:ask=4]')

2024-02-12 12:26:44 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-02-12 12:26:44 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        --- <exception caught here> ---
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
            error.trap(RemoteAmpError)
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/snap/maas/32469/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
        twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=320092:cmd=ReportNeighbours:ask=bbd1]')

```