Fuel for OpenStack

Wrong value passed from astute to nailgun 'virtual_sync_node'

Bug #1549323 reported by Alexandr Kostrikov on 2016-02-24

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Vladimir Sharshov	Fuel for OpenStack 9.0

Bug Description

At https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.network_templates/23/console there was an error:

2016-02-24 00:56:04,893 - ERROR __init__.py:67 -- assert_task_success raised: AssertionError("Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Critical nodes failed: Node[1], Node[3], Node[2], Node[5], Node[4]. Stopping the deployment process!'",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.network_templates/fuelweb_test/models/fuel_web_client.py", line 322, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Critical nodes failed: Node[1], Node[3], Node[2], Node[5], Node[4]. Stopping the deployment process!'

That happened because of errror:
[7f0b7b578740] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 57, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 273, in deploy_resp
    db_nodes = objects.NodeCollection.lock_for_update(q_nodes).all()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all
    return list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) invalid input syntax for integer: "virtual_sync_node"
LINE 3: WHERE nodes.id IN ('virtual_sync_node') ORDER BY nodes.id AS...

We should pass integer value instead of virtual_sync_node.

Tags:

Ivan Ponomarev (ivanzipfer) on 2016-02-25

tags:	added: area-astute
Changed in fuel:
status:	New → Confirmed

Nastya Urlapova (aurlapova) on 2016-02-29

tags:

added: swarm-blocker

Dmitry Pyzhov (dpyzhov) on 2016-03-02

tags:

added: module-astute
removed: area-astute

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-03: Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/287921

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-09: Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/287921
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=745b2a0723252b7e2a3f5315849a4f17e69600a9
Submitter: Jenkins
Branch: master

commit 745b2a0723252b7e2a3f5315849a4f17e69600a9
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Mar 3 18:32:06 2016 +0300

Prevent to report status for non-physical nodes

    Astute use virtual_sync_node for task_deployment as point for sync.
    It also try to report status for tasks on such node to Nailgun,
    but Nailgun do not expect such report.

Change-Id: I341456574e37dce25aa686c2f582bb898fb1a4c1
Closes-Bug: #1549323

Changed in fuel:
status:	In Progress → Fix Committed

Alexandr Kostrikov (akostrikov-mirantis) on 2016-03-25

Changed in fuel:
status:	Fix Committed → Fix Released

Revision history for this message

Alexandr Kostrikov (akostrikov-mirantis) wrote on 2016-03-25:

Checked with https://product-ci.infra.mirantis.net/job/9.0.all/101/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Related fix proposed to fuel-astute (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/308409

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Related fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/308409
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=de96a4b4bddcb5ba2e15dc930340f3efdec06ae7
Submitter: Jenkins
Branch: master

commit de96a4b4bddcb5ba2e15dc930340f3efdec06ae7
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Wed Apr 20 18:20:43 2016 +0300

Send node report message only after convertion.

After change proxy reporter behavior: now Astute send all messages
and validation only warn about possible problems.

    But from this moment Astute do not convert node name virtual_sync_node
    to real null for some messages e.g. error about deployment.
    This change fix it.

Change-Id: I016c3a2847aae15e2166922a7a9e5c1cd8110a74
Related-Bug: #1549323

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-20: Related fix proposed to fuel-astute (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308509

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-04-21: Related fix merged to fuel-astute (stable/mitaka)

Reviewed: https://review.openstack.org/308509
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=1340b8e8e98ef1f890daabffb396a2f95808cb67
Submitter: Jenkins
Branch: stable/mitaka

commit 1340b8e8e98ef1f890daabffb396a2f95808cb67
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Wed Apr 20 18:20:43 2016 +0300

Send node report message only after convertion.

After change proxy reporter behavior: now Astute send all messages
and validation only warn about possible problems.

    But from this moment Astute do not convert node name virtual_sync_node
    to real null for some messages e.g. error about deployment.
    This change fix it.

    Change-Id: I016c3a2847aae15e2166922a7a9e5c1cd8110a74
    Related-Bug: #1549323
    (cherry picked from commit de96a4b4bddcb5ba2e15dc930340f3efdec06ae7)

tags:

added: in-stable-mitaka

Revision history for this message

Jordan Olin (d-jordan) wrote on 2016-04-22:

Download full text (3.4 KiB)

This is also occurs in Fuel 8.0 build 1485. I am running into it trying to deploy a simple two node installation.

From Fuel Master RPC Consumer Log:

2016-04-22 19:20:17 ERROR [7f3bd9ebe740] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 57, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 278, in deploy_resp
    objects.NodeCollection.lock_for_update(q_nodes).all()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all
    return list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) invalid input syntax for integer: "virtual_sync_node"
LINE 3: WHERE nodes.id IN ('virtual_sync_node') ORDER BY nodes.id AS...
                           ^
[SQL: 'SELECT nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.cluster_id AS nodes_cluster_id, nodes.group_id AS nodes_group_id, nodes.name AS nodes_name, nodes.status AS nodes_status, nodes.meta AS nodes_meta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.hostname AS nodes_hostname, nodes.manufacturer AS nodes_manufacturer, nodes.platform_name AS nodes_platform_name, nodes.kernel_params AS nodes_kernel_params, nodes.progress AS nodes_progress, nodes.os_platform AS nodes_os_platform, nodes.pending_addition AS nodes_pending_addition, nodes.pending_deletion AS nodes_pending_deletion, nodes.error_type AS nodes_error_type, nodes.error_msg AS nodes_error_msg, nodes.timestamp AS nodes_timestamp, nodes.online AS nodes_online, nodes.labels AS nodes_labels, nodes.roles AS nodes_roles, nodes.pending_roles AS nodes_pending_roles, nodes.primary_roles AS nodes_primary_roles, nodes.agent_checksum AS nodes_agent_checksum, nodes.replaced_deployment_info AS nodes_replaced...

This is also occurs in Fuel 8.0 build 1485.  I am running into it trying to deploy a simple two node installation.

From Fuel Master RPC Consumer Log:

2016-04-22 19:20:17	ERROR	[7f3bd9ebe740] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 57, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 278, in deploy_resp
    objects.NodeCollection.lock_for_update(q_nodes).all()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2399, in all
    return list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) invalid input syntax for integer: "virtual_sync_node"
LINE 3: WHERE nodes.id IN ('virtual_sync_node') ORDER BY nodes.id AS...
                           ^
 [SQL: 'SELECT nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.cluster_id AS nodes_cluster_id, nodes.group_id AS nodes_group_id, nodes.name AS nodes_name, nodes.status AS nodes_status, nodes.meta AS nodes_meta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.hostname AS nodes_hostname, nodes.manufacturer AS nodes_manufacturer, nodes.platform_name AS nodes_platform_name, nodes.kernel_params AS nodes_kernel_params, nodes.progress AS nodes_progress, nodes.os_platform AS nodes_os_platform, nodes.pending_addition AS nodes_pending_addition, nodes.pending_deletion AS nodes_pending_deletion, nodes.error_type AS nodes_error_type, nodes.error_msg AS nodes_error_msg, nodes.timestamp AS nodes_timestamp, nodes.online AS nodes_online, nodes.labels AS nodes_labels, nodes.roles AS nodes_roles, nodes.pending_roles AS nodes_pending_roles, nodes.primary_roles AS nodes_primary_roles, nodes.agent_checksum AS nodes_agent_checksum, nodes.replaced_deployment_info AS nodes_replaced_deployment_info, nodes.replaced_provisioning_info AS nodes_replaced_provisioning_info, nodes.network_template AS nodes_network_template, nodes.extensions AS nodes_extensions \nFROM nodes \nWHERE nodes.id IN (%(id_1)s) ORDER BY nodes.id ASC FOR UPDATE'] [parameters: {'id_1': u'virtual_sync_node'}]

Revision history for this message

Jeffrey Guan (double12gzh) wrote on 2017-03-15:

Download full text (3.4 KiB)

Also found in fuel 9.0
2017-03-15 08:05:57 ERROR [7feccfb87740] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiverd.py", line 57, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/receiver.py", line 279, in deploy_resp
    db_nodes = objects.NodeCollection.lock_for_update(q_nodes).all()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all
    return list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) invalid input syntax for integer: "virtual_sync_node"
LINE 3: WHERE nodes.id IN ('virtual_sync_node') ORDER BY nodes.id AS...
                           ^
[SQL: 'SELECT nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.cluster_id AS nodes_cluster_id, nodes.group_id AS nodes_group_id, nodes.name AS nodes_name, nodes.status AS nodes_status, nodes.meta AS nodes_meta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.hostname AS nodes_hostname, nodes.manufacturer AS nodes_manufacturer, nodes.platform_name AS nodes_platform_name, nodes.kernel_params AS nodes_kernel_params, nodes.progress AS nodes_progress, nodes.os_platform AS nodes_os_platform, nodes.pending_addition AS nodes_pending_addition, nodes.pending_deletion AS nodes_pending_deletion, nodes.error_type AS nodes_error_type, nodes.error_msg AS nodes_error_msg, nodes.timestamp AS nodes_timestamp, nodes.online AS nodes_online, nodes.labels AS nodes_labels, nodes.roles AS nodes_roles, nodes.pending_roles AS nodes_pending_roles, nodes.primary_roles AS nodes_primary_roles, nodes.agent_checksum AS nodes_agent_checksum, nodes.replaced_deployment_info AS nodes_replaced_deployment_info, nodes.replaced_provisioning_info AS nodes_replaced_provisioning_info, nodes.network_tem...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.