MAAS regiond IPC crash due to a machine-resources binary crash when parsing some VPDs

[Dmitrii Shcherbakov]

This bug is a result of https://github.com/lxc/lxd/issues/10627.

Upon upgrading to 3.2 maas-regiond began failing to start an IPC worker since it tries to get a set of network devices present on a system and fails while calling out to a machine-resources binary.

It only happens due to some device VPDs which break the parser but the devices I had were fairly standard onboard NICs from Intel. lspci is able to parse the VPD of those devices and so should the parser in LXD.

After fixing the parser MAAS needs to use new binaries and so package rebuilds are needed.

2022-06-30 18:38:19 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:20.10:ppc64el:ga-20.10: to_add=['20212022-06-30 18:38:18 twisted.internet.defer: [critical] [513/5515]
        Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
            return target() File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task() File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task() --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 202, in wrapper result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 559, in retrier return func(*args, **kwargs)
          File "/usr/lib/python3.8/contextlib.py", line 75, in inner return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 361, in create_endpoints self._updateEndpoints(process, self._getListenAddresses(port))
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 303, in _getListenAddresses addresses = get_all_interface_source_addresses()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1207, in get_all_interface_source_addresses for network in get_all_interface_subnets():
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1192, in get_all_interface_subnets for interface in get_all_interfaces_definition().values()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1139, in get_all_interfaces_definition for name, ipaddr in get_ip_addr().items()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/ipaddr.py", line 28, in get_ip_addr output = call_and_check(command)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/shell.py", line 106, in call_and_check raise ExternalProcessError(process.returncode, command, output=stderr)
        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:

        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:
        panic: runtime error: slice bounds out of range [65535:32611]
        goroutine 1 [running]: github.com/lxc/lxd/lxd/resources.parsePCIVPD({0xc000b6a000, 0x8000, 0x8001})
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci_vpd.go:116 +0x312 github.com/lxc/lxd/lxd/resources.GetPCI()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci.go:146 +0x9e5 github.com/lxc/lxd/lxd/resources.GetResources()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/resources.go:48 +0x267 host-info/pkg/info.GetResources(...)
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:157 host-info/pkg/info.GetInfo()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:182 +0x34
        main.main()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/cmd/machine-resources/main.go:23 +0x1d

2022-06-30 18:38:18 twisted.internet.defer: [critical] Unhandled error in Deferred:
2022-06-30 18:38:18 twisted.internet.defer: [critical]
        Traceback (most recent call last):
          File "/usr/lib/python3.8/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 202, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 559, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.8/contextlib.py", line 75, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 361, in create_endpoints
            self._updateEndpoints(process, self._getListenAddresses(port))
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 303, in _getListenAddresses
            addresses = get_all_interface_source_addresses()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1207, in get_all_interface_source_addresses
            for network in get_all_interface_subnets():
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1192, in get_all_interface_subnets
            for interface in get_all_interfaces_definition().values()

          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1139, in get_all_interfaces_definition [309/6775] for name, ipaddr in get_ip_addr().items()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/ipaddr.py", line 28, in get_ip_addr output = call_and_check(command)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/shell.py", line 106, in call_and_check raise ExternalProcessError(process.returncode, command, output=stderr)
        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:
        panic: runtime error: slice bounds out of range [65535:32611]
        goroutine 1 [running]: github.com/lxc/lxd/lxd/resources.parsePCIVPD({0xc000d06000, 0x8000, 0x8001})
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci_vpd.go:116 +0x312
        github.com/lxc/lxd/lxd/resources.GetPCI()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci.go:146 +0x9e5
        github.com/lxc/lxd/lxd/resources.GetResources()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/resources.go:48 +0x267
        host-info/pkg/info.GetResources(...)
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:157
        host-info/pkg/info.GetInfo()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:182 +0x34
        main.main()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/cmd/machine-resources/main.go:23 +0x1d

[Anton Troyanov (troyanov)]

Just to have a link with bad VPDs here: https://github.com/lxc/lxd/files/9028036/bad-vpds.zip

[Dmitrii Shcherbakov (dmitriis)]

There's another case where the parser continues to fail but the previous one got resolved with the new LXD code (tested with binaries from https://launchpad.net/~troyanov/+archive/ubuntu/maas/+build/24165501).

https://github.com/lxc/lxd/issues/10662

[Anton Troyanov (troyanov)]

https://launchpad.net/~troyanov/+archive/ubuntu/maas/+build/24167488 contains a fresh version of lxd package where https://github.com/lxc/lxd/issues/10662 is fixed.

[Fridtjof Mund (fridtjofmund)]

Another case where the parser crashed for me: https://github.com/lxc/lxd/issues/10708