[2.1] expected string or bytes-like object

[Andres Rodriguez]

When accessing MAAS UI:

I see:

Error ocurred

expected string or bytes-like object

With these errors in the log:

2016-09-24 18:39:55 [-] Error on request (24) discovery.list: expected string or bytes-like object
        Traceback (most recent call last):
          File "/usr/lib/python3.5/threading.py", line 862, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 794, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 827, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.5/contextlib.py", line 30, in inner
            return func(*args, **kwds)
          File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 346, in list
            "%s__gt" % self._meta.batch_key: params["start"]
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 679, in filter
            return self._filter_or_exclude(False, *args, **kwargs)
          File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 697, in _filter_or_exclude
            clone.query.add_q(Q(*args, **kwargs))
          File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1310, in add_q
            clause, require_inner = self._add_q(where_part, self.used_aliases)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1338, in _add_q
            allow_joins=allow_joins, split_subq=split_subq,
          File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1209, in build_filter
            condition = self.build_lookup(lookups, col, value)
          File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1102, in build_lookup
            return final_lookup(lhs, rhs)
          File "/usr/lib/python3/dist-packages/django/db/models/lookups.py", line 101, in __init__
            self.rhs = self.get_prep_lookup()
          File "/usr/lib/python3/dist-packages/django/db/models/lookups.py", line 139, in get_prep_lookup
            return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
          File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 1313, in get_prep_lookup
            return super(DateField, self).get_prep_lookup(lookup_type, value)
          File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 727, in get_prep_lookup
            return self.get_prep_value(value)
          File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 1461, in get_prep_value
            value = super(DateTimeField, self).get_prep_value(value)
          File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 1317, in get_prep_value
            return self.to_python(value)
          File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 1420, in to_python
            parsed = parse_datetime(value)
          File "/usr/lib/python3/dist-packages/django/utils/dateparse.py", line 93, in parse_datetime
            match = datetime_re.match(value)
        builtins.TypeError: expected string or bytes-like object

[Mike Pontillo (mpontillo)]

If this happens again, we'll need the contents of the discovery view. (You can get that with "sudo maas-region dbshell" and then "select * from maasserver_discovery;".)

It seems like the only time this should ever happen is if the database returns an invalid (or NULL?) value for one of the datetime fields in the view. But those should never be missing, so it's hard to say why this would be happening.

[Nathaniel W. Turner (nturner)]

This reproduces reliably for me, so here's the output from the requested DB query.

[Mike Pontillo (mpontillo)]

@nturner, thanks.

What's causing this still seems to be a mystery. It seems like sometimes Django gets back a type from the database that it isn't expecting, and it's not clear why (given that all the fields in question should be not-null timestamp-with-time-zone fields). And looking at the code in Django, it seems to handle NULL values just fine. So perhaps it's getting back the wrong type.

I'm considering a change such as this to harden the view by making the data more consistent, but I'm still confused as to why this is happening in the first place. The SQL dump you provided looked normal.

http://paste.ubuntu.com/23273624/

Trying this patch is a little scary (depending on the environment), because you would need to shut down MAAS completely and run "sudo maas-region dbupgrade" for it to take effect.

[Mike Pontillo (mpontillo)]

I missed the fact this was only occurring when a batched query was requested over the websocket. So this issue only arises when there are a large amount of discoveries.

Note that the CLI/API still works if you want to list the discoveries, but the dashboard page is unfortunately broken.

I have a branch with a workaround that seems to work, which simply disables batched queries for network discoveries. A full solution (batch loading for database objects without auto-incrementing primary keys) is trickier and will require more effort. (we have a solution for machines using system_id, but system_id is a string that Django has no trouble comparing, whereas discoveries were using a first_seen time, and the comparison query was failing for some reason.