db json errors after icehouse juno upgrade

[Steve Baker]

from Don Waterloo http://www.gossamer-threads.com/lists/openstack/dev/42063

After doing this upgrade (with a good chunk of data in the database that was created on icehouse), i'm running into a problem I'm wondering if anyone else has seen.

In particular, if i do a 'heat resource-list' on a stack that was created with icehouse, i get an error, which is below.

I'm not sure how to approach this, my assumption is there is some data in my database which is not formatted correctly, but 'which' data is the problem.

I turned on mysql logging, and all the queries function, so it is parsing something that comes back which is the problem. The below trace makes me thing some column is null or '' in the old, and is expected in the new.

Any suggestions as to where to look next?

  File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 348, in __iter__
    return iter(self.resources)
  File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 159, in resources
    self.t.resource_definitions(self).items())
  File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 158, in <genexpr>
    for (name, data) in
  File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 191, in __init__
    resource = stack.db_resource_get(name)
  File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 190, in db_resource_get
    self.context, self.id)
  File "/usr/lib/python2.7/dist-packages/heat/db/api.py", line 96, in resource_get_all_by_stack
    return IMPL.resource_get_all_by_stack(context, stack_id)
  File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/api.py", line 263, in resource_get_all_by_stack
    options(orm.joinedload("data")).all()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2300, in all
    return list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 73, in instances
    rows = [process[0](row, None) for row in fetch]
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 455, in _instance
    populate_state(state, dict_, row, isnew, only_load_props)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 305, in populate_state
    populator(state, dict_, row)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 169, in fetch_col
    dict_[key] = row[col]
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/type_api.py", line 911, in process
    return process_value(impl_processor(value), dialect)
  File "/usr/lib/python2.7/dist-packages/heat/db/sqlalchemy/types.py", line 37, in process_result_value
    return loads(value)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/129750

[Don Bowman (donbowman)]

I ran:
update resource set properties_data = '{}' where properties_data is null;
on the heat database. I had 1200 null entries here from the icehouse usage.
And now the problem is gone.

So thank you, this was indeed my problem.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/129750
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9b792fb6a37fbe050d30b20f164aa6d28c8e2bd6
Submitter: Jenkins
Branch: master

commit 9b792fb6a37fbe050d30b20f164aa6d28c8e2bd6
Author: Steve Baker <email address hidden>
Date: Tue Oct 21 11:49:10 2014 +1300

    Tolerate null on Json db load

    A icehouse->juno migration issue has surfaced where
    resource.properties_data is null for existing resource records.

    This change fixes the Json db type to treat None the same as
    the serialized JSON 'null'.

    Change-Id: Ia0b9a23a7798c95fbbcba439ffb8203b50367e7c
    Closes-Bug: #1383505

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/130426

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130426
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3f7d4bb4adc35c0194d10b50aff00a5a919b2a32
Submitter: Jenkins
Branch: stable/juno

commit 3f7d4bb4adc35c0194d10b50aff00a5a919b2a32
Author: Steve Baker <email address hidden>
Date: Tue Oct 21 11:49:10 2014 +1300

    Tolerate null on Json db load

    A icehouse->juno migration issue has surfaced where
    resource.properties_data is null for existing resource records.

    This change fixes the Json db type to treat None the same as
    the serialized JSON 'null'.

    Change-Id: Ia0b9a23a7798c95fbbcba439ffb8203b50367e7c
    Closes-Bug: #1383505