Updates job failing on incorrect Heat request

[Ben Nemec]

I've seen the updates job fail on this a number of times. The error message we get back is:

ERROR: The server could not comply with the request since it is either malformed or otherwise incorrect.

See an example at http://logs.openstack.org/13/386713/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-updates/c2c4ec6/console.html#_2016-11-10_00_19_39_033868 until that log expires.

[Ben Nemec (bnemec)]

This looks like it may actually be a bug in Heat. I'm not sure what's going on, but it seems like an error reading from the database.

ERROR heat.common.wsgi [req-5cf05301-f072-4df7-a64a-a5ed48029dc0 d57c95b010064f78a42dcb9ca11d7ff6 6773cf3e3ee74a78a3d150083fa93ae6 - default default] Exception handling resource: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
    res = self.dispatcher.dispatch(message)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 225, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 195, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 424, in wrapped
    return func(self, ctx, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1714, in list_events
    stack_identifiers = {s.id: s.identifier() for s in stacks}

  File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1714, in <dictcomp>
    stack_identifiers = {s.id: s.identifier() for s in stacks}

  File "/usr/lib/python2.7/site-packages/heat/objects/stack.py", line 135, in get_all
    yield cls._from_db_object(context, cls(context), db_stack)

  File "/usr/lib/python2.7/site-packages/heat/objects/stack.py", line 73, in _from_db_object
    db_stack['raw_template']))

  File "/usr/lib/python2.7/site-packages/heat/objects/raw_template.py", line 57, in from_db_object
    tpl[field] = db_tpl[field]

TypeError: 'NoneType' object has no attribute '__getitem__'

[Ben Nemec (bnemec)]

This is the full log file that the traceback came from: http://logs.openstack.org/13/386713/3/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-updates/e1aa96a/logs/undercloud/var/log/heat/heat-api.txt.gz

[Thomas Herve (therve)]

See bug #1634127 and bug #1640466, looks like a duplicate too?

[Ben Nemec (bnemec)]

Ah, yeah sorry. Steve's bug didn't show up since it was marked as a duplicate. I'll just drop Heat from this one and leave it open on TripleO for tracking purposes.

[Crag Wolfe (cwolfe)]

Just curious, was there a "heat-manage purge_deleted" command in the mix somewhere? I'm guessing not. While https://review.openstack.org/#/c/388188/1/heat/objects/stack.py would avoid surfacing this issue, I still think we have a lurking issue. Something like: one thread updates a stack, marks it as completed and the old raw_template gets deleted while thread 2 is busy loading the stack and all of a sudden the raw_template is gone. One way around this could be to soft delete raw_templates in purge_db [1] rather than hard deleting them.

[1] https://review.openstack.org/gitweb?p=openstack/heat.git;a=blob;f=heat/engine/stack.py;h=3e3bd2caa49602a33eae8380e925f0eba302c7ae;hb=b3441c4a2af57ad713bf664f9a30d793840f749a#l2049

[Emilien Macchi (emilienm)]

This bug was last updated over 180 days ago, as tripleo is a fast moving project and we'd like to get the tracker down to currently actionable bugs, this is getting marked as Invalid. If the issue still exists, please feel free to reopen it.