taskflow

"Data too long for column 'meta' at row 1" in flowdetails table

Bug #1926304 reported by Gregory Thiemonge on 2021-04-27

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	taskflow	Fix Released	Undecided	Unassigned

Bug Description

Reported for Octavia in https://storyboard.openstack.org/#!/story/2008859

Octavia tests trigger some errors when using jobboard:

Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.StorageFailure: Failed updating flow details with uuid 'e3321117-0c02-4049-869d-120f9623b884'
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: sqlalchemy.exc.DataError: (pymysql.err.DataError) (1406, "Data too long for column 'meta' at row 1")
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: [SQL: UPDATE flowdetails SET updated_at=%(updated_at)s, meta=%(meta)s, name=%(name)s, state=%(state)s, uuid=%(uuid)s WHERE flowdetails.uuid = %(uuid_1)s]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: [parameters: {'updated_at': datetime.datetime(2021, 4, 26, 23, 41, 58, 701021), 'meta': '{"factory": {"name": "octavia.controller.worker.v2.flows.flow_utils.get_cascade_delete_load_balancer_flow", "args": [{"admin_state_up": true, "descri ... (76598 characters truncated) ... abled": false, "project_id": "9a000ef1c0fa4bf8b243acd98976148f", "tls_versions": null, "tls_ciphers": null, "alpn_protocols": null}]], "kwargs": {}}}', 'name': 'get_cascade_delete_load_balancer_flow-e3321117-0c02-4049-869d-120f9623b884', 'state': None, 'uuid': 'e3321117-0c02-4049-869d-120f9623b884', 'uuid_1': 'e3321117-0c02-4049-869d-120f9623b884'}]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: (Background on this error at: http://sqlalche.me/e/13/9h9h)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server cursor, statement, parameters, context
Apr 26 23:41:58 devstack1 octavia-worker[1648795]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server result = self._query(query)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server conn.query(q)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server result.read()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server first_packet = self.connection._read_packet()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server packet.raise_for_error()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server pymysql.err.DataError: (1406, "Data too long for column 'meta' at row 1")
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/taskflow/persistence/backends/impl_sqlalchemy.py", line 483, in update_flow_details
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server self._update_flow_details(conn, flow_detail, e_fd)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/taskflow/persistence/backends/impl_sqlalchemy.py", line 462, in _update_flow_details
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server .values(e_fd.to_dict()))
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server return meth(self, multiparams, params)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(self, multiparams, params)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server distilled_params,
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server e, statement, parameters, cursor, context
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server sqlalchemy_exception, with_traceback=exc_info[2], from_=e
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server raise exception
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server cursor, statement, parameters, context
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server result = self._query(query)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server conn.query(q)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server result.read()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server first_packet = self.connection._read_packet()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server packet.raise_for_error()
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server sqlalchemy.exc.DataError: (pymysql.err.DataError) (1406, "Data too long for column 'meta' at row 1")
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server [SQL: UPDATE flowdetails SET updated_at=%(updated_at)s, meta=%(meta)s, name=%(name)s, state=%(state)s, uuid=%(uuid)s WHERE flowdetails.uuid = %(uuid_1)s]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server [parameters: {'updated_at': datetime.datetime(2021, 4, 26, 23, 41, 58, 701021), 'meta': '{"factory": {"name": "octavia.controller.worker.v2.flows.flow_utils.get_cascade_delete_load_balancer_flow", "args": [{"admin_state_up": true, "descri ... (76598 characters truncated) ... abled": false, "project_id": "9a000ef1c0fa4bf8b243acd98976148f", "tls_versions": null, "tls_ciphers": null, "alpn_protocols": null}]], "kwargs": {}}}', 'name': 'get_cascade_delete_load_balancer_flow-e3321117-0c02-4049-869d-120f9623b884', 'state': None, 'uuid': 'e3321117-0c02-4049-869d-120f9623b884', 'uuid_1': 'e3321117-0c02-4049-869d-120f9623b884'}]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server (Background on this error at: http://sqlalche.me/e/13/9h9h)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/opt/stack/octavia/octavia/controller/queue/v2/endpoints.py", line 56, in delete_load_balancer
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server self.worker.delete_load_balancer(loadbalancer, cascade)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/opt/stack/octavia/octavia/controller/worker/v2/controller_worker.py", line 390, in delete_load_balancer
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server load_balancer, listeners, pools, store=store)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/opt/stack/octavia/octavia/controller/worker/v2/controller_worker.py", line 103, in run_flow
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server self.services_controller.run_poster(func, *args, **kwargs)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/opt/stack/octavia/octavia/common/base_taskflow.py", line 143, in run_poster
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server backend=persistence)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/taskflow/engines/helpers.py", line 201, in save_factory_details
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server conn.update_flow_details(flow_detail)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/taskflow/persistence/backends/impl_sqlalchemy.py", line 488, in update_flow_details
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server " uuid '%s'" % flow_detail.uuid)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/taskflow/exceptions.py", line 52, in raise_with_cause
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 142, in raise_with_cause
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server raise exc_cls(message, *args, **kwargs) from kwargs.get('cause')
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server taskflow.exceptions.StorageFailure: Failed updating flow details with uuid 'e3321117-0c02-4049-869d-120f9623b884'
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server sqlalchemy.exc.DataError: (pymysql.err.DataError) (1406, "Data too long for column 'meta' at row 1")
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server [SQL: UPDATE flowdetails SET updated_at=%(updated_at)s, meta=%(meta)s, name=%(name)s, state=%(state)s, uuid=%(uuid)s WHERE flowdetails.uuid = %(uuid_1)s]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server [parameters: {'updated_at': datetime.datetime(2021, 4, 26, 23, 41, 58, 701021), 'meta': '{"factory": {"name": "octavia.controller.worker.v2.flows.flow_utils.get_cascade_delete_load_balancer_flow", "args": [{"admin_state_up": true, "descri ... (76598 characters truncated) ... abled": false, "project_id": "9a000ef1c0fa4bf8b243acd98976148f", "tls_versions": null, "tls_ciphers": null, "alpn_protocols": null}]], "kwargs": {}}}', 'name': 'get_cascade_delete_load_balancer_flow-e3321117-0c02-4049-869d-120f9623b884', 'state': None, 'uuid': 'e3321117-0c02-4049-869d-120f9623b884', 'uuid_1': 'e3321117-0c02-4049-869d-120f9623b884'}]
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server (Background on this error at: http://sqlalche.me/e/13/9h9h)
Apr 26 23:41:58 devstack1 octavia-worker[1648794]: ERROR oslo_messaging.rpc.server

Note that the meta field in the flowdetails table is a "text" data type:

$ mysql -u root octavia_persistence -e 'describe flowdetails'

the text data type is limited to 64kbytes.
taskflow defines 'meta' as a JSON doc (mariadb and mysql document that the JSON data type is the same size as the LONGTEXT data type).

Updating meta to a longtext type fixes the issue:

> ALTER TABLE flowdetails MODIFY meta LONGTEXT;

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-27: Fix proposed to taskflow (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/taskflow/+/788328

Changed in taskflow:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-01: Fix merged to taskflow (master)

Reviewed: https://review.opendev.org/c/openstack/taskflow/+/788328
Committed: https://opendev.org/openstack/taskflow/commit/ebdb7a85d0cbd84578e9b48f7ac4c2dea94a3798
Submitter: "Zuul (22348)"
Branch: master

commit ebdb7a85d0cbd84578e9b48f7ac4c2dea94a3798
Author: Gregory Thiemonge <email address hidden>
Date: Tue Apr 27 16:53:20 2021 +0200

Fix flowdetails meta size

    meta field in flowdetails is defined as a JSON data type, but its data
    type is 'text' in mysql, which is limited to 64kbytes. JSON data type
    should have the same size as a LONGTEXT.

Closes-Bug: #1926304

Change-Id: I9f89badfc697f0f26245ca7f4c22d62e220be5f9

Changed in taskflow:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-01: Fix included in openstack/taskflow 4.6.1

This issue was fixed in the openstack/taskflow 4.6.1 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.