many retries can overflow the results column in atomdetails

Bug #1416088 reported by Greg Hill
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
taskflow
Fix Released
High
Greg Hill

Bug Description

If you have a task that retries a lot of times (up to 80 in this particular case, which I know is excessive, but makes sense in context), it can eventually write a JSON string to the database that is longer than the field can handle. In the default mysql backend, mysql will silently truncate that JSON string, making it no longer valid. Any further retries then fail, because they can't load the history due to it being invalid JSON.

For example (elided for sanity):

Original exception being dropped: ['Traceback (most recent call last):\n', ..File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode\n obj, end = self.scan_once(s, idx)\n', 'ValueError: Unterminated string starting at: line 1 column 65497 (char 65497)\n']

Greg Hill (greg-hill)
description: updated
Joshua Harlow (harlowja)
Changed in taskflow:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to taskflow (master)

Reviewed: https://review.openstack.org/153443
Committed: https://git.openstack.org/cgit/openstack/taskflow/commit/?id=1fc1a7e4713c27f537a3f6192bea259a0d63ca64
Submitter: Jenkins
Branch: master

commit 1fc1a7e4713c27f537a3f6192bea259a0d63ca64
Author: Joshua Harlow <email address hidden>
Date: Thu Feb 5 17:46:58 2015 -0800

    Add warning to sqlalchemy backend size limit docs

    Partial-Bug: #1416088

    Change-Id: I21bde00445892a4c734592cbcc143f23085e5660

Revision history for this message
Joshua Harlow (harlowja) wrote :

So I also just tried this:

http://paste.openstack.org/show/198192/

The usage of 'mysql_sql_mode': 'TRADITIONAL', there will avoid saving the truncated data.

With the mysql I am running on this causes the following exception on save.

taskflow.exceptions.StorageFailure: Failed updating flow details with uuid '193754c4-bf08-415c-9172-a54fec0bd690'
  DataError: (DataError) (1406, "Data too long for column 'meta' at row 1") 'UPDATE flowdetails....

Greg Hill (greg-hill)
Changed in taskflow:
assignee: nobody → Greg Hill (greg-hill)
Revision history for this message
Ann Taraday (akamyshnikova) wrote :

I guess this is fixed now as part of https://bugs.launchpad.net/taskflow/+bug/1838015

Changed in taskflow:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.