Dumpdata failing for table metadataserver.nodeuserdata

[Ivan Hitos]

1) Output from dpkg -l '*maas*'|cat

root@maas-server:~# dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==============================-============-=================================================
ii maas 2.2.0+bzr6054-0ubuntu1~16.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-proxy 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.2.0+bzr6054-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.2.0+bzr6054-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.2.0+bzr6054-0ubuntu1~16.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.2.0+bzr6054-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)

-------------------------------------------

2) Instructions on how we can re-create your bug and what the expected result should be.

Please see command executed and output:

sudo maas-region dumpdata metadataserver.nodeuserdata > file.json
CommandError: Unable to serialize database: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte. You passed in b'\x1f\x8b\x08\x00\x00\ <snippet> \xe0QR[\x10\x00\x00' (<class 'metadataserver.fields.Bin'>)

A select or a pg_dump to that table works fine.

[Doug Parrish (dparrish)]

This occurs after MAAS deploys a node for Juju. The "data" field for the row contains base64-encoded gzip data which contains:
#cloud-config
apt_mirror: ""
output:
  all: '| tee -a /var/log/cloud-init-output.log'
package_update: true
package_upgrade: true
packages:
- bridge-utils
- curl
- cpu-checker
.
.
.

[Mike Pontillo (mpontillo)]

Can I ask what you are using this command for? This is a Django command that just happens to exist because we didn't take steps to block its usage. It is neither tested or supported in the context of MAAS, to my knowledge.

[Aydin Doyak (aydintd)]

We were thinking about using these below commands for collecting MAAS data from customers for support purposes in case of any need:

1. To get information about MAAS network diagnostics, MAAS' cluster configuration, bootable images and their metadata information :

sudo maas-rack support-dump > maas-support.dump

I did not come across any problems while running above command.

2. To get MAAS database dump and/or it's tables dump in particular in JSON format for investigation :

sudo maas-region dumpdata > `hostname -f`-maas-db-dump.json (to get full dump)

or

sudo maas-region dumpdata metadataserver.nodeuserdata > file.json (to get a spesific table's dump in JSON format)

which fails when trying to get a db dump in JSON format if MAAS deploys a node for Juju. Error message is present in the bug description. This prevents us to collect dump data from MAAS if it's used for deploying a node via juju.

[David A. Desrosiers (setuid)]

I recently ran into the same exact issue with MAAS + dumpdata, and determined that it was Juju (2.3.2 and 2.3.3) that triggeredg dumpdata to fail, while a "normal" machine deployment did not. This doesn't mean Juju is necessarily at fault, it just triggers this behavior.

Running dumpdata with --traceback, we get:

$ sudo maas-region dumpdata --traceback 1>/dev/null
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/utils/encoding.py", line 88, in force_text
    s = six.text_type(s, encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/maas-region", line 65, in <module>
    main()
  File "/usr/sbin/maas-region", line 61, in main
    run()
  File "/usr/sbin/maas-region", line 49, in run
    management.execute_from_command_line()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/django/core/management/commands/dumpdata.py", line 162, in handle
    stream=stream or self.stdout)
  File "/usr/lib/python3/dist-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/usr/lib/python3/dist-packages/django/core/serializers/base.py", line 61, in serialize
    self.handle_field(obj, field)
  File "/usr/lib/python3/dist-packages/django/core/serializers/python.py", line 55, in handle_field
    self._current[field.name] = field.value_to_string(obj)
  File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 852, in value_to_string
    return smart_text(self._get_val_from_obj(obj))
  File "/usr/lib/python3/dist-packages/django/utils/encoding.py", line 56, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/usr/lib/python3/dist-packages/django/utils/encoding.py", line 103, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
django.utils.encoding.DjangoUnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte.

After tracing down the issue this week, I found that the line in question that breaks dumpdata here, is line 88 in '/usr/lib/python3/dist-packages/django/utils/encoding.py'.

This very simple, 1-line patch appears to fix the issue, so dumpdata works with the expected results, without damaging the intent:

--- /tmp/encoding.orig 2018-02-23 10:54:35.159622665 -0500
+++ /tmp/encoding.new 2018-02-23 10:54:27.711520872 -0500
@@ -85,7 +85,7 @@
         if not isinstance(s, six.string_types):
             if six.PY3:
                 if isinstance(s, byte...