Adding 20.04 images to MAAS causes UI tracebacks for TypeError: '<' not supported between instances of 'NoneType' and 'str'

[Jeff Lane ]

This is seen on 2.6 installs:
MAAS version: 2.6.1 (7832-g17912cdc9-0ubuntu1~18.04.1)
MAAS version: 2.6.0~beta4 (7743-g27b04aaf3-0ubuntu1~18.04.1)

Two different MAAS servers with 2.6ish versions of MAAS.

I noticed today that 20.04 images are available for syncing, so I selected that on both maas servers and synced.

Once the images were downloaded, MAAS started throwing tracebacks in regiond.log and the UI now does not work.

On the UI, I get this:
Error occurredReload
'<' not supported between instances of 'NoneType' and 'str'

and in regiond.log I get this:
2019-11-05 16:33:54 maasserver.websockets.protocol: [critical] Error on request (55153) general.osinfo: '<' not supported between instances of 'NoneType' and 'str'
        Traceback (most recent call last):
          File "/usr/lib/python3.6/threading.py", line 864, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 861, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 894, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 235, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 563, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.6/contextlib.py", line 52, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 397, in prep_user_execute
            return method(params)
          File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/general.py", line 100, in osinfo
            releases = list_all_usable_releases()
          File "/usr/lib/python3/dist-packages/maasserver/utils/osystems.py", line 82, in list_all_usable_releases
            distro_series[osystem] = sorted(releases, key=itemgetter('title'))
        builtins.TypeError: '<' not supported between instances of 'NoneType' and 'str'

[Jeff Lane  (bladernr)]

Additionally, I'm trying to figure out how to remove boot images, however, the documentation only covers how to create these, not remove them:

https://maas.io/docs/cli-image-management

So I'm left with two broken MAAS environments and no way (that I can see) to resolve this via CLI.

[Sean Feole (sfeole)]

I am able to confirm this is also happening on our production MAAS servers. After trying to import 20.04 images today.
I was able to workaround this by restarting maas-regiond and maas-rackd

sudo service maas-regiond restart
sudo service maas-rackd restart

It appears that MAAS is working fine and the images are available to deploy with. But i have not tried it yet.

[Alberto Donato (ack)]

This is likely happening because 20.04 is not found in distro-info-data.

Updating the package and restarting maas should workaround the issue until it's properly fixed.

[David A. Desrosiers (setuid)]

I was just hit by this yesterday, and again today, using:

MAAS version: 2.6.0 (7802-g59416a869-0ubuntu1~18.10.1)

I can confirm that restarting 'maas-regiond' and 'maas-rackd' does not fix the issue.

Removing my local cached copy of the simplestreams images and re-sync'ing them does not fix the issue.

Pointing to upstream images.maas.io and re-sync'ing the images, does not fix the issue.

(re)installing 'distro-info-data' does not fix the issue.

Oddly, using either a local mirror or upstream copy of images, shows '20.04' in the list, until the image is sync'd, then the UI when refreshed, shows 'focal' as the name in the image list (see linked image[0]).

The only thing that worked to resolve this, was to check my 'boot-resources', then delete each one that matched 'focal', using 'boot-resource delete XX' where 'XX' is the number of the imported-and-cached boot image in MAAS.

[0] https://www.dropbox.com/s/wrqakz2wk4x2shx/Screenshot_2019-11-20_07-58-18.png

[Sean Feole (sfeole)]

Hey David,
I also have MAAS 2.6.0

I just hit this issue again by installing Focal on my MAAS. Please ensure that distro-info-data is updated to the latest.

distro-info-data:
  Installed: 0.37ubuntu0.6
  Candidate: 0.37ubuntu0.6

I had to update my pkg archives to reveal there was an update. (apt update)

After installing 0.6 then restart maas region & rackd

[dann frazier (dannf)]

I hit this this morning just after importing hirsute images. Our MAAS UI is currently unusable. Seemingly correlated with this, only the MAAS frontpage seems to load - all others spin in "Loading..."

[dann frazier (dannf)]

installed: 2.8.2-8577-g.a3e674063 (8980) 140MB -

2020-11-11 15:15:42 maasserver.websockets.protocol: [critical] Error on request (2) config.list: '<' not supported between instances of 'NoneType' and 'str'
 Traceback (most recent call last):
   File "/snap/maas/8980/usr/lib/python3.6/threading.py", line 864, in run
     self._target(*self._args, **self._kwargs)
   File "/snap/maas/8980/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 850, in worker
     return target()
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
     result = inContext.theWork()
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/snap/maas/8980/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
     return func(*args,**kw)
   File "/snap/maas/8980/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 885, in callInContext
     return func(*args, **kwargs)
   File "/snap/maas/8980/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 217, in wrapper
     result = func(*args, **kwargs)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/utils/orm.py", line 737, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/utils/orm.py", line 540, in retrier
     return func(*args, **kwargs)
   File "/snap/maas/8980/usr/lib/python3.6/contextlib.py", line 52, in inner
     return func(*args, **kwds)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/websockets/base.py", line 405, in prep_user_execute
     method_name, method, params
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/websockets/base.py", line 428, in _call_method_track_queries
     result = method(params)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/websockets/handlers/config.py", line 68, in list
     return self._include_choices(config_keys)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/websockets/handlers/config.py", line 47, in _include_choices
     self._include_choice(config_key)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/websockets/handlers/config.py", line 38, in _include_choice
     config_field = get_config_field(config_key["name"])
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/forms/settings.py", line 931, in get_config_field
     return conf["form"](**kwargs)
   File "/snap/maas/8980/lib/python3.6/site-packages/maasserver/forms/settings.py", line 64, in make...

[dann frazier (dannf)]

The issue immediately recovered after removing all hirsute images via the CLI.

[dann frazier (dannf)]

Since I hadn't deselected syncing of the hirsute images, the issue quickly returned. I again deleted them, the UI became responsive again, then remove hirsute images from my image selection.

[Bruno Eduardo Bernardi (bruno-bernardi22)]

Hello Dann Frazier,

We have the same problem in our maas, but we use it via SNAP and we were unable to use the client to remove the image and our MAAS UI is currently unusable.

Could you please help us on how the image was removed from your chest?

Thanks in advance!

[dann frazier (dannf)]

@Bruno: We are also using the snap.

What I did was use the MAAS CLI to list all the images, and hunted down the IDs of the hirsute ones:

$ maas login maas http://ip.of.maas.server/MAAS
$ maas maas boot-resources read | grep -A23 hirsute

Mine were IDs 231-235. I deleted them like this:
$ for i in $(seq 231 235); do maas maas boot-resource delete ${i}; done

[Bruno Eduardo Bernardi (bruno-bernardi22)]

Hello Dann Frazier,

This Work's! Thank you so much for the solution!

Best regards.