[CDH] Cluster Scale up fails on Update configs

Bug #1493543 reported by Peter Nordquist
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Sahara
Fix Released
High
Vitalii Gridnev
Kilo
Fix Released
Medium
Vitalii Gridnev
Liberty
Fix Released
High
Vitalii Gridnev

Bug Description

When I attempt to scale up a cluster that uses the CDH 5.3.0 plugin, it fails during the 'Update configs' step. I've included the failed event below:

{
                "successful": null,
                "tenant_id": "<tenant_id>",
                "created_at": "2015-09-01T00:22:37",
                "step_type": "Plugin: scale cluster",
                "updated_at": null,
                "cluster_id": "<cluster_id>",
                "step_name": "Update configs",
                "total": 1,
                "id": "<id>"
},

From my research into this, I found that the section linked here https://github.com/openstack/sahara/blob/stable/kilo/sahara/plugins/cdh/cloudera_utils.py#L139-L155 is where the failure is occurring. This piece of code is using a thread pool to invoke the cloudera manager to deploy the client config. The thread pool is re-wrapping the exception so the real stacktrace never makes it into the sahara logs. The reason code I find in the logs is '__exit__'.

Tags: plugin.cdh
Changed in sahara:
status: New → Confirmed
assignee: nobody → Vitaly Gridnev (vgridnev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (master)

Fix proposed to branch: master
Review: https://review.openstack.org/225656

Changed in sahara:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to sahara (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/225657

tags: added: plugin.cdh
Changed in sahara:
importance: Undecided → High
milestone: none → liberty-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/225656
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=55224cd84eb4d7680484dc156384fc6d67e2cf09
Submitter: Jenkins
Branch: master

commit 55224cd84eb4d7680484dc156384fc6d67e2cf09
Author: Vitaly Gridnev <email address hidden>
Date: Mon Sep 21 15:41:40 2015 +0300

    Fix wrong init of ThreadGrop

    ThreadGroup is an object, and should be initialized.
    Closes-bug: 1493543

    Change-Id: If4b074efd87c226ee219aa2e3cfe458cadbbfb81

Changed in sahara:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (stable/kilo)

Reviewed: https://review.openstack.org/225657
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=51397f3992313587846603cd33229abe20687342
Submitter: Jenkins
Branch: stable/kilo

commit 51397f3992313587846603cd33229abe20687342
Author: Vitaly Gridnev <email address hidden>
Date: Mon Sep 21 15:41:40 2015 +0300

    Fix wrong init of ThreadGrop

    ThreadGroup is an object, and should be initialized.
    Closes-bug: 1493543

    Change-Id: If4b074efd87c226ee219aa2e3cfe458cadbbfb81

Thierry Carrez (ttx)
Changed in sahara:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in sahara:
milestone: liberty-rc1 → 3.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.