incorrect application parameter:"dfs.replication" for scaling cluster

[Vadim Rovachev]

Step 1.
Create cluster:
By Example:
    with topology:
        {[namenode, jobtracker]x1,
        [secondarynamenode]x1,
        [datanode, tasktracker]x2,
        [datanode]x1,
        [tasktracker]x1}
    and with configuration:
        ["cluster_configs]["HDFS"]["dfs.replication": 2]

Step 2.
Check the application parameters:
Go to namenode, upload file in HDFS and write command 'hadoop fsck /<path_to_file> -files -blocks -locations | grep "Default replication factor:"'
expected result:
    By Example:
         Default replication factor: 2

Step 3.
Scale cluster with changes:
    delete [datanode]
    delete [tasktracker]
    add [other_datanode]
    add [other_tasktracker]
    resize [datanode, tasktracker], count: 2 -> 3
to topology:
    {[namenode, jobtracker]x1,
    [secondarynamenode]x1,
    [datanode, tasktracker]x3,
    [other_datanode]x1,
    [other_tasktracker]x1}

expected result:
    scale cluster
actual result:
13-09-02 19:40:51.966 9172 ERROR savanna.utils.api [-] Request aborted with status code 500 and message 'Internal Server Error'
2013-09-02 19:40:51.966 9172 ERROR savanna.utils.api [-] Traceback (most recent call last):
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/utils/api.py", line 81, in handler
    return func(**kwargs)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/service/validation.py", line 79, in handler
    return func(*args, **kwargs)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/service/validation.py", line 50, in handler
    return func(*args, **kwargs)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/api/v10.py", line 49, in clusters_scale
    return u.render(api.scale_cluster(cluster_id, data).wrapped_dict)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 55, in scale_cluster
    context.model_update(cluster, status='Active')
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 52, in scale_cluster
    plugin.validate_scaling(cluster, to_be_enlarged, additional)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/plugins/vanilla/plugin.py", line 175, in validate_scaling
    self._validate_existing_ng_scaling(cluster, existing)
  File "/home/vadim/savanna/fromSF/savanna/.tox/venv/local/lib/python2.7/site-packages/savanna/plugins/vanilla/plugin.py", line 318, in _validate_existing_ng_scaling
    "(replication factor is %s )" % rep_factor)
Exception: Vanilla plugin cannot shrink cluster because it would be not enough nodes for replicas (replication factor is 3 )

127.0.0.1 - - [02/Sep/2013 19:40:51] "PUT /v1.0/4c18f28bf72843a39d77a5645422c099/clusters/70f439d0-20e7-48c1-80b9-8444687972d8 HTTP/1.1" 500 228 0.374602

----------------------------------------------------------------And more------------------------------------------------------------------------
Step 1.
Create cluster with 4 worker nodes with topology ['tasktracker', 'datanode'] and with configuration: ["cluster_configs]["HDFS"]["dfs.replication": 4]

Step 2.
Scale cluster with changes:
resize ['tasktracker', 'datanode'] 4 -> 3

expected result:
    Exception: Vanilla plugin cannot shrink cluster because it would be not enough nodes for replicas (replication factor is 4 )
actual result:
    Scale cluster

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47191

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47191
Committed: http://github.com/stackforge/savanna/commit/fb9acb1d904c3ab022a145546a504fd97b07d9f2
Submitter: Jenkins
Branch: master

commit fb9acb1d904c3ab022a145546a504fd97b07d9f2
Author: Andrew Lazarev <email address hidden>
Date: Wed Sep 18 19:44:03 2013 +0400

    Fixed rep_factor calculation in cluster shrink validation

    Fixes: bug #1219903

    Change-Id: I6a51fc2698b2a598849ab684c0266a1888342938