Deploy 30 nodes ambari cluster.
sahara-engine.log
2017-11-27 10:25:10.787 21 ERROR sahara.service.ops [req-0cc035b8-b13e-402e-914e-b627d78322e2 f8dc89ef2f9940b8a9a7f15e141e1d9e 0b8ee214c1604b69a031ae8b549e2797 - - -] [instance: none, cluster: bd02c4d7-c5ab-4a68-b738-0b7784fbef96] Error during operating on cluster (reason: An error occurred in thread 'configure-instance-ideadata-zd-303v1-ambari-slave-3t-24': (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT anon_1.clusters_created_at AS anon_1_clusters_created_at, anon_1.clusters_updated_at AS anon_1_clusters_updated_at, anon_1.clusters_id AS anon_1_clusters_id, anon_1.clusters_name AS anon_1_clusters_name, anon_1.clusters_description AS anon_1_clusters_description, anon_1.clusters_tenant_id AS anon_1_clusters_tenant_id, anon_1.clusters_trust_id AS anon_1_clusters_trust_id, anon_1.clusters_is_transient AS anon_1_clusters_is_transient, anon_1.clusters_plugin_name AS anon_1_clusters_plugin_name, anon_1.clusters_hadoop_version AS anon_1_clusters_hadoop_version, anon_1.clusters_cluster_configs AS anon_1_clusters_cluster_configs, anon_1.clusters_default_image_id AS anon_1_clusters_default_image_id, anon_1.clusters_neutron_management_network AS anon_1_clusters_neutron_management_network, anon_1.clusters_anti_affinity AS anon_1_clusters_anti_affinity, anon_1.clusters_management_private_key AS anon_1_clusters_management_private_key, anon_1.clusters_management_public_key AS anon_1_clusters_management_public_key, anon_1.clusters_user_keypair_id AS anon_1_clusters_user_keypair_id, anon_1.clusters_status AS anon_1_clusters_status, anon_1.clusters_status_description AS anon_1_clusters_status_description, anon_1.clusters_info AS anon_1_clusters_info, anon_1.clusters_extra AS anon_1_clusters_extra, anon_1.clusters_rollback_info AS anon_1_clusters_rollback_info, anon_1.clusters_sahara_info AS anon_1_clusters_sahara_info, anon_1.clusters_use_autoconfig AS anon_1_clusters_use_autoconfig, anon_1.clusters_cluster_template_id AS anon_1_clusters_cluster_template_id, anon_1.clusters_shares AS anon_1_clusters_shares, anon_1.clusters_is_public AS anon_1_clusters_is_public, anon_1.clusters_is_protected AS anon_1_clusters_is_protected, anon_1.clusters_domain_name AS anon_1_clusters_domain_name, cluster_events_1.created_at AS cluster_events_1_created_at, cluster_events_1.updated_at AS cluster_events_1_updated_at, cluster_events_1.id AS cluster_events_1_id, cluster_events_1.node_group_id AS cluster_events_1_node_group_id, cluster_events_1.instance_id AS cluster_events_1_instance_id, cluster_events_1.instance_name AS cluster_events_1_instance_name, cluster_events_1.event_info AS cluster_events_1_event_info, cluster_events_1.successful AS cluster_events_1_successful, cluster_events_1.step_id AS cluster_events_1_step_id, cluster_provision_steps_1.created_at AS cluster_provision_steps_1_created_at, cluster_provision_steps_1.updated_at AS cluster_provision_steps_1_updated_at, cluster_provision_steps_1.id AS cluster_provision_steps_1_id, cluster_provision_steps_1.cluster_id AS cluster_provision_steps_1_cluster_id, cluster_provision_steps_1.tenant_id AS cluster_provision_steps_1_tenant_id, cluster_provision_steps_1.step_name AS cluster_provision_steps_1_step_name, cluster_provision_steps_1.step_type AS cluster_provision_steps_1_step_type, cluster_provision_steps_1.total AS cluster_provision_steps_1_total, cluster_provision_steps_1.successful AS cluster_provision_steps_1_successful, cluster_health_checks_1.created_at AS cluster_health_checks_1_created_at, cluster_health_checks_1.updated_at AS cluster_health_checks_1_updated_at, cluster_health_checks_1.id AS cluster_health_checks_1_id, cluster_health_checks_1.verification_id AS cluster_health_checks_1_verification_id, cluster_health_checks_1.status AS cluster_health_checks_1_status, cluster_health_checks_1.description AS cluster_health_checks_1_description, cluster_health_checks_1.name AS cluster_health_checks_1_name, cluster_verifications_1.created_at AS cluster_verifications_1_created_at, cluster_verifications_1.updated_at AS cluster_verifications_1_updated_at, cluster_verifications_1.id AS cluster_verifications_1_id, cluster_verifications_1.cluster_id AS cluster_verifications_1_cluster_id, cluster_verifications_1.status AS cluster_verifications_1_status, instances_1.created_at AS instances_1_created_at, instances_1.updated_at AS instances_1_updated_at, instances_1.id AS instances_1_id, instances_1.tenant_id AS instances_1_tenant_id, instances_1.node_group_id AS instances_1_node_group_id, instances_1.instance_id AS instances_1_instance_id, instances_1.instance_name AS instances_1_instance_name, instances_1.internal_ip AS instances_1_internal_ip, instances_1.management_ip AS instances_1_management_ip, instances_1.volumes AS instances_1_volumes, instances_1.storage_devices_number AS instances_1_storage_devices_number, instances_1.dns_hostname AS instances_1_dns_hostname, node_group_templates_1.created_at AS node_group_templates_1_created_at, node_group_templates_1.updated_at AS node_group_templates_1_updated_at, node_group_templates_1.id AS node_group_templates_1_id, node_group_templates_1.name AS node_group_templates_1_name, node_group_templates_1.description AS node_group_templates_1_description, node_group_templates_1.tenant_id AS node_group_templates_1_tenant_id, node_group_templates_1.flavor_id AS node_group_templates_1_flavor_id, node_group_templates_1.image_id AS node_group_templates_1_image_id, node_group_templates_1.plugin_name AS node_group_templates_1_plugin_name, node_group_templates_1.hadoop_version AS node_group_templates_1_hadoop_version, node_group_templates_1.node_processes AS node_group_templates_1_node_processes, node_group_templates_1.node_configs AS node_group_templates_1_node_configs, node_group_templates_1.volumes_per_node AS node_group_templates_1_volumes_per_node, node_group_templates_1.volumes_size AS node_group_templates_1_volumes_size, node_group_templates_1.volumes_availability_zone AS node_group_templates_1_volumes_availability_zone, node_group_templates_1.volume_mount_prefix AS node_group_templates_1_volume_mount_prefix, node_group_templates_1.volume_type AS node_group_templates_1_volume_type, node_group_templates_1.floating_ip_pool AS node_group_templates_1_floating_ip_pool, node_group_templates_1.security_groups AS node_group_templates_1_security_groups, node_group_templates_1.auto_security_group AS node_group_templates_1_auto_security_group, node_group_templates_1.availability_zone AS node_group_templates_1_availability_zone, node_group_templates_1.is_proxy_gateway AS node_group_templates_1_is_proxy_gateway, node_group_templates_1.volume_local_to_instance AS node_group_templates_1_volume_local_to_instance, node_group_templates_1.is_default AS node_group_templates_1_is_default, node_group_templates_1.use_autoconfig AS node_group_templates_1_use_autoconfig, node_group_templates_1.shares AS node_group_templates_1_shares, node_group_templates_1.is_public AS node_group_templates_1_is_public, node_group_templates_1.is_protected AS node_group_templates_1_is_protected, node_groups_1.created_at AS node_groups_1_created_at, node_groups_1.updated_at AS node_groups_1_updated_at, node_groups_1.id AS node_groups_1_id, node_groups_1.name AS node_groups_1_name, node_groups_1.tenant_id AS node_groups_1_tenant_id, node_groups_1.flavor_id AS node_groups_1_flavor_id, node_groups_1.image_id AS node_groups_1_image_id, node_groups_1.image_username AS node_groups_1_image_username, node_groups_1.node_processes AS node_groups_1_node_processes, node_groups_1.node_configs AS node_groups_1_node_configs, node_groups_1.volumes_per_node AS node_groups_1_volumes_per_node, node_groups_1.volumes_size AS node_groups_1_volumes_size, node_groups_1.volumes_availability_zone AS node_groups_1_volumes_availability_zone, node_groups_1.volume_mount_prefix AS node_groups_1_volume_mount_prefix, node_groups_1.volume_type AS node_groups_1_volume_type, node_groups_1.count AS node_groups_1_count, node_groups_1.use_autoconfig AS node_groups_1_use_autoconfig, node_groups_1.cluster_id AS node_groups_1_cluster_id, node_groups_1.node_group_template_id AS node_groups_1_node_group_template_id, node_groups_1.floating_ip_pool AS node_groups_1_floating_ip_pool, node_groups_1.security_groups AS node_groups_1_security_groups, node_groups_1.auto_security_group AS node_groups_1_auto_security_group, node_groups_1.availability_zone AS node_groups_1_availability_zone, node_groups_1.open_ports AS node_groups_1_open_ports, node_groups_1.is_proxy_gateway AS node_groups_1_is_proxy_gateway, node_groups_1.volume_local_to_instance AS node_groups_1_volume_local_to_instance, node_groups_1.shares AS node_groups_1_shares, node_group_templates_2.created_at AS node_group_templates_2_created_at, node_group_templates_2.updated_at AS node_group_templates_2_updated_at, node_group_templates_2.id AS node_group_templates_2_id, node_group_templates_2.name AS node_group_templates_2_name, node_group_templates_2.description AS node_group_templates_2_description, node_group_templates_2.tenant_id AS node_group_templates_2_tenant_id, node_group_templates_2.flavor_id AS node_group_templates_2_flavor_id, node_group_templates_2.image_id AS node_group_templates_2_image_id, node_group_templates_2.plugin_name AS node_group_templates_2_plugin_name, node_group_templates_2.hadoop_version AS node_group_templates_2_hadoop_version, node_group_templates_2.node_processes AS node_group_templates_2_node_processes, node_group_templates_2.node_configs AS node_group_templates_2_node_configs, node_group_templates_2.volumes_per_node AS node_group_templates_2_volumes_per_node, node_group_templates_2.volumes_size AS node_group_templates_2_volumes_size, node_group_templates_2.volumes_availability_zone AS node_group_templates_2_volumes_availability_zone, node_group_templates_2.volume_mount_prefix AS node_group_templates_2_volume_mount_prefix, node_group_templates_2.volume_type AS node_group_templates_2_volume_type, node_group_templates_2.floating_ip_pool AS node_group_templates_2_floating_ip_pool, node_group_templates_2.security_groups AS node_group_templates_2_security_groups, node_group_templates_2.auto_security_group AS node_group_templates_2_auto_security_group, node_group_templates_2.availability_zone AS node_group_templates_2_availability_zone, node_group_templates_2.is_proxy_gateway AS node_group_templates_2_is_proxy_gateway, node_group_templates_2.volume_local_to_instance AS node_group_templates_2_volume_local_to_instance, node_group_templates_2.is_default AS node_group_templates_2_is_default, node_group_templates_2.use_autoconfig AS node_group_templates_2_use_autoconfig, node_group_templates_2.shares AS node_group_templates_2_shares, node_group_templates_2.is_public AS node_group_templates_2_is_public, node_group_templates_2.is_protected AS node_group_templates_2_is_protected, templates_relations_1.created_at AS templates_relations_1_created_at, templates_relations_1.updated_at AS templates_relations_1_updated_at, templates_relations_1.id AS templates_relations_1_id, templates_relations_1.tenant_id AS templates_relations_1_tenant_id, templates_relations_1.name AS templates_relations_1_name, templates_relations_1.flavor_id AS templates_relations_1_flavor_id, templates_relations_1.image_id AS templates_relations_1_image_id, templates_relations_1.node_processes AS templates_relations_1_node_processes, templates_relations_1.node_configs AS templates_relations_1_node_configs, templates_relations_1.volumes_per_node AS templates_relations_1_volumes_per_node, templates_relations_1.volumes_size AS templates_relations_1_volumes_size, templates_relations_1.volumes_availability_zone AS templates_relations_1_volumes_availability_zone, templates_relations_1.volume_mount_prefix AS templates_relations_1_volume_mount_prefix, templates_relations_1.volume_type AS templates_relations_1_volume_type, templates_relations_1.count AS templates_relations_1_count, templates_relations_1.use_autoconfig AS templates_relations_1_use_autoconfig, templates_relations_1.cluster_template_id AS templates_relations_1_cluster_template_id, templates_relations_1.node_group_template_id AS templates_relations_1_node_group_template_id, templates_relations_1.floating_ip_pool AS templates_relations_1_floating_ip_pool, templates_relations_1.security_groups AS templates_relations_1_security_groups, templates_relations_1.auto_security_group AS templates_relations_1_auto_security_group, templates_relations_1.availability_zone AS templates_relations_1_availability_zone, templates_relations_1.is_proxy_gateway AS templates_relations_1_is_proxy_gateway, templates_relations_1.volume_local_to_instance AS templates_relations_1_volume_local_to_instance, templates_relations_1.shares AS templates_relations_1_shares, cluster_templates_1.created_at AS cluster_templates_1_created_at, cluster_templates_1.updated_at AS cluster_templates_1_updated_at, cluster_templates_1.id AS cluster_templates_1_id, cluster_templates_1.name AS cluster_templates_1_name, cluster_templates_1.description AS cluster_templates_1_description, cluster_templates_1.cluster_configs AS cluster_templates_1_cluster_configs, cluster_templates_1.default_image_id AS cluster_templates_1_default_image_id, cluster_templates_1.anti_affinity AS cluster_templates_1_anti_affinity, cluster_templates_1.tenant_id AS cluster_templates_1_tenant_id, cluster_templates_1.neutron_management_network AS cluster_templates_1_neutron_management_network, cluster_templates_1.plugin_name AS cluster_templates_1_plugin_name, cluster_templates_1.hadoop_version AS cluster_templates_1_hadoop_version, cluster_templates_1.is_default AS cluster_templates_1_is_default, cluster_templates_1.use_autoconfig AS cluster_templates_1_use_autoconfig, cluster_templates_1.shares AS cluster_templates_1_shares, cluster_templates_1.is_public AS cluster_templates_1_is_public, cluster_templates_1.is_protected AS cluster_templates_1_is_protected, cluster_templates_1.domain_name AS cluster_templates_1_domain_name \nFROM (SELECT clusters.created_at AS clusters_created_at, clusters.updated_at AS clusters_updated_at, clusters.id AS clusters_id, clusters.name AS clusters_name, clusters.description AS clusters_description, clusters.tenant_id AS clusters_tenant_id, clusters.trust_id AS clusters_trust_id, clusters.is_transient AS clusters_is_transient, clusters.plugin_name AS clusters_plugin_name, clusters.hadoop_version AS clusters_hadoop_version, clusters.cluster_configs AS clusters_cluster_configs, clusters.default_image_id AS clusters_default_image_id, clusters.neutron_management_network AS clusters_neutron_management_network, clusters.anti_affinity AS clusters_anti_affinity, clusters.management_private_key AS clusters_management_private_key, clusters.management_public_key AS clusters_management_public_key, clusters.user_keypair_id AS clusters_user_keypair_id, clusters.status AS clusters_status, clusters.status_description AS clusters_status_description, clusters.info AS clusters_info, clusters.extra AS clusters_extra, clusters.rollback_info AS clusters_rollback_info, clusters.sahara_info AS clusters_sahara_info, clusters.use_autoconfig AS clusters_use_autoconfig, clusters.cluster_template_id AS clusters_cluster_template_id, clusters.shares AS clusters_shares, clusters.is_public AS clusters_is_public, clusters.is_protected AS clusters_is_protected, clusters.domain_name AS clusters_domain_name \nFROM clusters \nWHERE (clusters.tenant_id = %(tenant_id_1)s OR clusters.is_public = 1) AND clusters.id = %(id_1)s \n LIMIT %(param_1)s) AS anon_1 LEFT OUTER JOIN cluster_provision_steps AS cluster_provision_steps_1 ON anon_1.clusters_id = cluster_provision_steps_1.cluster_id LEFT OUTER JOIN cluster_events AS cluster_events_1 ON cluster_provision_steps_1.id = cluster_events_1.step_id LEFT OUTER JOIN cluster_verifications AS cluster_verifications_1 ON anon_1.clusters_id = cluster_verifications_1.cluster_id LEFT OUTER JOIN cluster_health_checks AS cluster_health_checks_1 ON cluster_verifications_1.id = cluster_health_checks_1.verification_id LEFT OUTER JOIN node_groups AS node_groups_1 ON anon_1.clusters_id = node_groups_1.cluster_id LEFT OUTER JOIN instances AS instances_1 ON node_groups_1.id = instances_1.node_group_id LEFT OUTER JOIN node_group_templates AS node_group_templates_1 ON node_group_templates_1.id = node_groups_1.node_group_template_id LEFT OUTER JOIN cluster_templates AS cluster_templates_1 ON cluster_templates_1.id = anon_1.clusters_cluster_template_id LEFT OUTER JOIN templates_relations AS templates_relations_1 ON cluster_templates_1.id = templates_relations_1.cluster_template_id LEFT OUTER JOIN node_group_templates AS node_group_templates_2 ON node_group_templates_2.id = templates_relations_1.node_group_template_id ORDER BY instances_1.instance_name'] [parameters: {u'id_1': u'bd02c4d7-c5ab-4a68-b738-0b7784fbef96', u'param_1': 1, u'tenant_id_1': u'0b8ee214c1604b69a031ae8b549e2797'}]
Traceback (most recent call last):
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/context.py", line 168, in _wrapper
func(*args, **kwargs)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/utils/cluster_progress_ops.py", line 128, in handler
if not cluster_utils.check_cluster_exists(cluster_id):
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/utils/cluster.py", line 116, in check_cluster_exists
cluster = conductor.cluster_get(ctx, cluster)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/conductor/resource.py", line 42, in handle
ret = func(*args, **kwargs)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/conductor/api.py", line 65, in cluster_get
context, _get_id(cluster), show_progress)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/conductor/manager.py", line 162, in cluster_get
return self.db.cluster_get(context, cluster, show_progress)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/db/api.py", line 117, in cluster_get
cluster = IMPL.cluster_get(context, cluster)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/db/sqlalchemy/api.py", line 290, in cluster_get
return _cluster_get(context, get_session(), cluster_id)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sahara/db/sqlalchemy/api.py", line 286, in _cluster_get
return query.filter_by(id=cluster_id).first()
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2659, in first
ret = list(self[0:1])
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
return list(res)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
return self._execute_and_instances(context)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
util.raise_from_cause(newraise, exc_info)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 835, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1019, in _read_query_result
result.read()
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1309, in read
self._read_result_packet(first_packet)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1372, in _read_result_packet
self._read_rowdata_packet()
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1406, in _read_rowdata_packet
packet = self.connection._read_packet()
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 971, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_bytes
2013, "Lost connection to MySQL server during query")
Can you please share more details about the environment? Please specify:
- how the environment was configured (apparently kolla is involved, but it is not clear exactly how)
- which version (master?)
- the information about the cluster that it's being created
(that said, I strongly suspect that the issue is on the way kolla configured the environment, but I could be wrong).