Error during the integration CDH test

[Denis Egorenko]

During the integration CDH test some error occurs:

FAIL: tests.gating.test_cdh_gating.CDHGatingTest.test_cdh_plugin_gating[cdh]
tags: worker-0
----------------------------------------------------------------------
stderr: {{{
INFO [paramiko.transport] Connected (version 2.0, client OpenSSH_5.3)
    INFO [paramiko.transport] Authentication (publickey) successful!
    INFO [paramiko.transport] Secsh channel 1 opened.
    INFO [paramiko.transport] Secsh channel 2 opened.
}}}

stdout: {{{
!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!* ERROR LOG *!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!

Failure while cluster creation: invalid literal for int() with base 10: ''

!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!* END OF ERROR LOG *!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*!
}}}

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-sahara-integration-cdh/sahara/tests/integration/tests/gating/test_cdh_gating.py", line 326, in test_cdh_plugin_gating
    self._create_cluster()
  File "sahara/tests/integration/tests/base.py", line 49, in wrapper
    ITestCase.print_error_log(message, e)
  File "sahara/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "sahara/tests/integration/tests/base.py", line 46, in wrapper
    fct(*args, **kwargs)
  File "/home/jenkins/workspace/gate-sahara-integration-cdh/sahara/tests/integration/tests/gating/test_cdh_gating.py", line 187, in _create_cluster
    self.cdh_config)
  File "sahara/tests/integration/tests/base.py", line 341, in await_active_workers_for_namenode
    active_datanode_count = int(active_datanode_count)
ValueError: invalid literal for int() with base 10: ''
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 2 tests in 966.400s
FAILED (id=0, failures=2)
error: testr failed (1)
ERROR: InvocationError: '/home/jenkins/workspace/gate-sahara-integration-cdh/.tox/integration/bin/python setup.py test --slowest --testr-args=cdh --concurrency=1'
___________________________________ summary ____________________________________
ERROR: integration: commands failed

So, the reason is wrong check for active data nodes:
(https://github.com/openstack/sahara/blob/master/sahara/tests/integration/tests/base.py#L338)

'sudo -u %s bash -lc "hadoop dfsadmin -report" | '
                        'grep "Datanodes available:.*" | awk \'{print $3}\''

As a result we have empty parameter.

Output for command `hadoop dfsadmin -report` doesn't contains words `Datanodes available`. Output contains next:

Configured Capacity: 59051573248 (55.00 GB)
Present Capacity: 59051622400 (55.00 GB)
DFS Remaining: 58878752735 (54.84 GB)
DFS Used: 172869665 (164.86 MB)
DFS Used%: 0.29%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 10.50.0.134:50010 (sahara-cdh-test-cdh-worker-node-nm-dn-002.nl)
Hostname: sahara-cdh-test-cdh-worker-node-nm-dn-002.nl
Rack: /default
Decommission Status : Normal
Configured Capacity: 29525786624 (27.50 GB)
DFS Used: 98038809 (93.50 MB)
Non DFS Used: 0 (0 B)
DFS Remaining: 29427747815 (27.41 GB)
DFS Used%: 0.33%
DFS Remaining%: 99.67%
Configured Cache Capacity: 4294967296 (4 GB)
Cache Used: 0 (0 B)
Cache Remaining: 4294967296 (4 GB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Last contact: Mon Aug 04 16:18:08 MSK 2014
-------------------------------------------------

So, i suggest to use another check, something like:
'sudo -u %s bash -lc "hadoop dfsadmin -report" | '
                        'grep "Live datanodes.*" | grep -o "[0-9]" '

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112534
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=3d5b2d0088423eeb1dbe482e99f6bdb293cfb3fc
Submitter: Jenkins
Branch: master

commit 3d5b2d0088423eeb1dbe482e99f6bdb293cfb3fc
Author: Sergey Reshetnyak <email address hidden>
Date: Thu Aug 7 13:34:55 2014 +0400

    Fix parsing dfsreport for CDH in integration tests

    Change-Id: I099425fc58e33d47a0a4424a9e16753f7cde064a
    Closes-bug: #1352325