Sahara

[CDH] TWO HBASE MASTER clusters don't go through creation

Bug #1656869 reported by Rafik HARZI on 2017-01-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Sahara	Fix Released	Low	Rafik HARZI	Sahara ocata-3 "o3"

Bug Description

Sahara does not validate the number of HBASE MASTER before CDH cluster creation But it raises an ERROR while configuring two HBASE MASTER processes.
To reproduce this bug create a cluster template that includes two HBASE MASTERS (be careful for dependencies with other processes)

Error log:
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [req-32f2ecec-4e99-4852-b7a7-254f3db453fb d3cbb9a869914b58a8b3b2cb0f0eb821 1b97b590a9374b2392c5db59511e6278 - - -] [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] Error during operating on cluster (reason: Hadoop cluster should contain 0 or 1 HBASE_MASTER component(s). Actual HBASE_MASTER count is 2
Error ID: b2d0436d-b744-42e2-97cf-b6f085f7b272)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] Traceback (most recent call last):
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/service/ops.py", line 192, in wrapper
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] f(cluster_id, *args, **kwds)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/service/ops.py", line 294, in _provision_cluster
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] plugin.configure_cluster(cluster)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/cdh/plugin.py", line 63, in configure_cluster
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] cluster.hadoop_version).configure_cluster(cluster)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/cdh/abstractversionhandler.py", line 102, in configure_cluster
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] self.deploy.configure_cluster(cluster)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/cdh/v5_5_0/deploy.py", line 81, in configure_cluster
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] CU.create_services(cluster)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/utils/cluster_progress_ops.py", line 139, in handler
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] add_fail_event(instance, e)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] self.force_reraise()
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] six.reraise(self.type_, self.value, self.tb)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/utils/cluster_progress_ops.py", line 136, in handler
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] value = func(*args, **kwargs)
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/cdh/v5_5_0/cloudera_utils.py", line 124, in create_services
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] if self.pu.get_hbase_master(cluster):
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/cdh/plugin_utils.py", line 156, in get_hbase_master
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] return u.get_instance(cluster, 'HBASE_MASTER')
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] File "/home/sahara/sahara/.venv/local/lib/python2.7/site-packages/sahara/plugins/utils.py", line 45, in get_instance
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] node_process, _('0 or 1'), len(instances))
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] InvalidComponentCountException: Hadoop cluster should contain 0 or 1 HBASE_MASTER component(s). Actual HBASE_MASTER count is 2
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e] Error ID: b2d0436d-b744-42e2-97cf-b6f085f7b272
2017-01-16 15:45:43.953 14693 ERROR sahara.service.ops [instance: none, cluster: 6e2ff30e-3023-403d-89c9-e77b6b6e0b6e]

Tags:

Vitalii Gridnev (vgridnev) on 2017-01-17

Changed in sahara:
status:	New → Confirmed
importance:	Undecided → Low

Revision history for this message

Vitalii Gridnev (vgridnev) wrote on 2017-01-17:

https://review.openstack.org/#/c/421157/

Changed in sahara:
status:	Confirmed → In Progress
assignee:	nobody → Vitaly Gridnev (vgridnev)
milestone:	none → ocata-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Change abandoned on sahara (master)

Change abandoned by Vitaly Gridnev (<email address hidden>) on branch: master
Review: https://review.openstack.org/421157

Rafik HARZI (rafik.harzi) on 2017-01-17

Changed in sahara:
assignee:	Vitaly Gridnev (vgridnev) → Rafik HARZI (rafik.harzi)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix merged to sahara (master)

Reviewed: https://review.openstack.org/421165
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=3d63bdc9b4cf1e305044d8f63047b47004188297
Submitter: Jenkins
Branch: master

commit 3d63bdc9b4cf1e305044d8f63047b47004188297
Author: Rafik <email address hidden>
Date: Mon Jan 16 16:49:17 2017 +0100

Add HBASE MASTER processes number validation

    Because of the missing HBASE MASTER process number validation
    during cluster creation, it takes too long for sahara to raise
    the error too many HBASE MASTER processes.

Closes-Bug: #1656869

Change-Id: Id143eb2360366443a630f65fc6261231b562417c