MAAS regiond IPC crash due to a machine-resources binary crash when parsing some VPDs

Bug #1980490 reported by Dmitrii Shcherbakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Committed
High
Anton Troyanov
3.2
Fix Released
High
Anton Troyanov
lxd
Fix Released
Unknown

Bug Description

This bug is a result of https://github.com/lxc/lxd/issues/10627.

Upon upgrading to 3.2 maas-regiond began failing to start an IPC worker since it tries to get a set of network devices present on a system and fails while calling out to a machine-resources binary.

It only happens due to some device VPDs which break the parser but the devices I had were fairly standard onboard NICs from Intel. lspci is able to parse the VPD of those devices and so should the parser in LXD.

After fixing the parser MAAS needs to use new binaries and so package rebuilds are needed.

2022-06-30 18:38:19 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:20.10:ppc64el:ga-20.10: to_add=['20212022-06-30 18:38:18 twisted.internet.defer: [critical] [513/5515]
        Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs) File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
            return target() File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task() File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task() --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 202, in wrapper result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 559, in retrier return func(*args, **kwargs)
          File "/usr/lib/python3.8/contextlib.py", line 75, in inner return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 361, in create_endpoints self._updateEndpoints(process, self._getListenAddresses(port))
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 303, in _getListenAddresses addresses = get_all_interface_source_addresses()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1207, in get_all_interface_source_addresses for network in get_all_interface_subnets():
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1192, in get_all_interface_subnets for interface in get_all_interfaces_definition().values()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1139, in get_all_interfaces_definition for name, ipaddr in get_ip_addr().items()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/ipaddr.py", line 28, in get_ip_addr output = call_and_check(command)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/shell.py", line 106, in call_and_check raise ExternalProcessError(process.returncode, command, output=stderr)
        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:

        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:
        panic: runtime error: slice bounds out of range [65535:32611]
        goroutine 1 [running]: github.com/lxc/lxd/lxd/resources.parsePCIVPD({0xc000b6a000, 0x8000, 0x8001})
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci_vpd.go:116 +0x312 github.com/lxc/lxd/lxd/resources.GetPCI()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci.go:146 +0x9e5 github.com/lxc/lxd/lxd/resources.GetResources()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/resources.go:48 +0x267 host-info/pkg/info.GetResources(...)
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:157 host-info/pkg/info.GetInfo()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:182 +0x34
        main.main()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/cmd/machine-resources/main.go:23 +0x1d

2022-06-30 18:38:18 twisted.internet.defer: [critical] Unhandled error in Deferred:
2022-06-30 18:38:18 twisted.internet.defer: [critical]
        Traceback (most recent call last):
          File "/usr/lib/python3.8/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 202, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 559, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.8/contextlib.py", line 75, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 361, in create_endpoints
            self._updateEndpoints(process, self._getListenAddresses(port))
          File "/usr/lib/python3/dist-packages/maasserver/ipc.py", line 303, in _getListenAddresses
            addresses = get_all_interface_source_addresses()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1207, in get_all_interface_source_addresses
            for network in get_all_interface_subnets():
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1192, in get_all_interface_subnets
            for interface in get_all_interfaces_definition().values()

          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1139, in get_all_interfaces_definition [309/6775] for name, ipaddr in get_ip_addr().items()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/ipaddr.py", line 28, in get_ip_addr output = call_and_check(command)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/shell.py", line 106, in call_and_check raise ExternalProcessError(process.returncode, command, output=stderr)
        provisioningserver.utils.shell.ExternalProcessError: Command `sudo /usr/share/maas/machine-resources/amd64` returned non-zero exit status 2:
        panic: runtime error: slice bounds out of range [65535:32611]
        goroutine 1 [running]: github.com/lxc/lxd/lxd/resources.parsePCIVPD({0xc000d06000, 0x8000, 0x8001})
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci_vpd.go:116 +0x312
        github.com/lxc/lxd/lxd/resources.GetPCI()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/pci.go:146 +0x9e5
        github.com/lxc/lxd/lxd/resources.GetResources()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/vendor/github.com/lxc/lxd/lxd/resources/resources.go:48 +0x267
        host-info/pkg/info.GetResources(...)
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:157
        host-info/pkg/info.GetInfo()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/pkg/info/info.go:182 +0x34
        main.main()
                /build/maas-QVndNG/maas-3.2.0-11989-g.84a255c14/src/host-info/cmd/machine-resources/main.go:23 +0x1d

Related branches

Alberto Donato (ack)
summary: - [3.2] MAAS regiond IPC crash due to a machine-resources binary crash
- when parsing some VPDs
+ MAAS regiond IPC crash due to a machine-resources binary crash when
+ parsing some VPDs
Changed in maas:
milestone: none → 3.3.0
importance: Undecided → High
status: New → Triaged
Revision history for this message
Anton Troyanov (troyanov) wrote :

Just to have a link with bad VPDs here: https://github.com/lxc/lxd/files/9028036/bad-vpds.zip

Changed in lxd:
status: Unknown → New
Changed in lxd:
status: New → Fix Released
Changed in maas:
status: Triaged → Fix Committed
Changed in maas:
assignee: nobody → Anton Troyanov (troyanov)
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

There's another case where the parser continues to fail but the previous one got resolved with the new LXD code (tested with binaries from https://launchpad.net/~troyanov/+archive/ubuntu/maas/+build/24165501).

https://github.com/lxc/lxd/issues/10662

Revision history for this message
Anton Troyanov (troyanov) wrote :
Changed in lxd:
status: Fix Released → Unknown
Revision history for this message
Fridtjof Mund (fridtjofmund) wrote :

Another case where the parser crashed for me: https://github.com/lxc/lxd/issues/10708

Changed in lxd:
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.