[nailgun] receiverd.py fails to update nodes error_msg

Bug #1578245 reported by Vasiliy Pleshakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Roman Prykhodchenko
Mitaka
Fix Released
High
Roman Prykhodchenko

Bug Description

Detailed bug description:
while debuging plugin in task based deployment mode I see next error repeated 3 times in receiverd.log:
2016-04-25 16:18:40.142 INFO [7f8155019740] (receiver) RPC method deploy_resp received: {"task_uuid": "17569447-4200-4669-9a2f-4f09c83fde48", "nodes": [{"status": "error", "uid": "11", "error_type": "deploy", "deployment_graph_task_name": "nsm", "progress": 100, "task_status": "error", "error_msg": "All nodes are finished. Failed tasks: Task[nsm/11], Task[puppet/11], Task[keepalived/11], Task[haproxy/11], Task[mariadb_galera/11], Task[puppet/10], Task[nsm/12], Task[puppet/12], Task[keepalived/12], Task[haproxy/12], Task[mariadb_galera/12], Task[puppet/7], Task[nsm/9], Task[puppet/9], Task[keepalived/9], Task[haproxy/9], Task[mariadb_galera/9] Stopping the deployment process!"}]}
2016-04-25 16:18:40.166 DEBUG [7f8155019740] (receiver) Updating node 11 - set error_msg to All nodes are finished. Failed tasks: Task[nsm/11], Task[puppet/11], Task[keepalived/11], Task[haproxy/11], Task[mariadb_galera/11], Task[puppet/10], Task[nsm/12], Task[puppet/12], Task[keepalived/12], Task[haproxy/12], Task[mariadb_galera/12], Task[puppet/7], Task[nsm/9], Task[puppet/9], Task[keepalived/9], Task[haproxy/9], Task[mariadb_galera/9] Stopping the deployment process!
2016-04-25 16:18:40.166 DEBUG [7f8155019740] (receiver) Updating node 11 - set error_type to deploy
2016-04-25 16:18:40.166 DEBUG [7f8155019740] (receiver) Updating node 11 - set status to error
2016-04-25 16:18:40.166 DEBUG [7f8155019740] (receiver) Updating node 11 - set progress to 100
2016-04-25 16:18:40.191 ERROR [7f8155019740] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 57, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 313, in deploy_resp
    task_uuid=task_uuid
  File "/usr/lib/python2.7/site-packages/nailgun/notifier.py", line 26, in notify
    "task_uuid": task_uuid
  File "/usr/lib/python2.7/site-packages/nailgun/objects/notification.py", line 60, in create
    task = Task.get_by_uuid(task_uuid)
  File "/usr/lib/python2.7/site-packages/nailgun/objects/task.py", line 68, in get_by_uuid
    res = q.first()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2634, in first
    ret = list(self[0:1])
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
    return list(res)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2735, in __iter__
    self.session._autoflush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1303, in _autoflush
    util.raise_from_cause(e)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1293, in _autoflush
    self.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2019, in flush
    self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2137, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2101, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 706, in _emit_update_statements
    execute(statement, multiparams)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.DataError) value too long for type character varying(255)
 [SQL: 'UPDATE nodes SET error_msg=%(error_msg)s WHERE nodes.id = %(nodes_id)s'] [parameters: {'nodes_id': 11, 'error_msg': u'All nodes are finished. Failed tasks: Task[nsm/11], Task[puppet/11], Task[keepalived/11], Task[haproxy/11], Task[mariadb_galera/11], Task[puppet/10], Task[nsm/12], Task[puppet/12], Task[keepalived/12], Task[haproxy/12], Task[mariadb_galera/12], Task[puppet/7], Task[nsm/9], Task[puppet/9], Task[keepalived/9], Task[haproxy/9], Task[mariadb_galera/9] Stopping the deployment process!'}]
Steps to reproduce:

Expected results:
 no tracebacks in logs.
Reproducibility:
 always.
Workaround:
 not found.
Description of the environment:
 Operation system: FUEL9 build 4010 on Centos7
 Versions of components: LCM Plugin
 Reference architecture: cluster with at least 1 controller and 1 LCM controller.
 Network model: Neutron with VLAN segmentation
 Related projects installed: LCM plugin https://gerrit.mirantis.com/#/q/project:att-lcm/lcm-plugin

Tags: area-python
Changed in fuel:
milestone: none → 9.0
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
importance: Undecided → Medium
milestone: 9.0 → 10.0
status: New → Confirmed
tags: added: area-python
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Roman Prykhodchenko (romcheg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/326390

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Bulat Gaifullin (bulat.gaifullin) wrote :

It is not medium. because feature does not work.

Changed in fuel:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326659

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/326390
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=78e45f1c0326f1adfed6cfe4919b63e26b8e423d
Submitter: Jenkins
Branch: master

commit 78e45f1c0326f1adfed6cfe4919b63e26b8e423d
Author: Roman Prykhodchenko <email address hidden>
Date: Tue Jun 7 13:21:10 2016 +0200

    Allow long error messages for nodes

    Change type of error message attribute for nodes
    from a 255 character long string, i.e., sa.String(255)
    to a text of variable lenght -- sa.Text.

    Change-Id: I5d1d83458d961302d4696ee5089197eacc420688
    Closes-bug: #1578245

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/326659
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=be6aab91c693cb6c6498d94d72500a318bd827e3
Submitter: Jenkins
Branch: stable/mitaka

commit be6aab91c693cb6c6498d94d72500a318bd827e3
Author: Roman Prykhodchenko <email address hidden>
Date: Tue Jun 7 13:21:10 2016 +0200

    Allow long error messages for nodes

    Change type of error message attribute for nodes
    from a 255 character long string, i.e., sa.String(255)
    to a text of variable lenght -- sa.Text.

    Change-Id: I5d1d83458d961302d4696ee5089197eacc420688
    Closes-bug: #1578245
    (cherry-picked from commit 78e45f1c0326f1adfed6cfe4919b63e26b8e423d)

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Verified on 9-#465

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.