Mirantis OpenStack

[Backport][bug/1351624]Error on starting services on cluster

Bug #1401787 reported by Alexander Ignatov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Committed
Critical
Alexander Ignatov
Mirantis OpenStack 5.1.1-updates

Bug Description

This bug is to backport fix for https://bugs.launchpad.net/sahara/+bug/1351624

==============
Original dscription
==============

I was using stable/icehouse and Vanilla plugin. When I provisioned a Hadoop cluster on a slow machine, the provisioning failed with the following exception:

DEBUG sahara.utils.ssh_remote [testcluster-master-001] Executing "sudo su - -c "hadoop-daemon.sh start namenode" hadoop" from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-master-001] _execute_command took 5.9 seconds to complete from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-master-001] Executing "sudo su - -c "yarn-daemon.sh start resourcemanager" hadoop" from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-master-001] _execute_command took 2.9 seconds to complete from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-worker-001] Executing "sudo su - -c "hadoop-daemon.sh start datanode" hadoop" from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-worker-001] _execute_command took 6.3 seconds to complete from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-worker-002] Executing "sudo su - -c "hadoop-daemon.sh start datanode" hadoop" from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.utils.ssh_remote [testcluster-worker-002] _execute_command took 5.6 seconds to complete from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
INFO sahara.plugins.vanilla.v2_3_0.run_scripts Waiting 2 datanodes to start up
DEBUG sahara.plugins.vanilla.v2_3_0.run_scripts Checking datanode count from (pid=17022) _check_datanodes_count /home/ubuntu/sahara/sahara/sahara/plugins/vanilla/v2_3_0/run_scripts.py:148
DEBUG sahara.utils.ssh_remote [testcluster-master-001] Executing "sudo su -lc "hadoop dfsadmin -report" hadoop | grep 'Datanodes available:' | awk '{print $3}'" from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.openstack.common.periodic_task Running periodic task SaharaPeriodicTasks.update_job_statuses from (pid=17022) run_periodic_tasks /home/ubuntu/sahara/sahara/sahara/openstack/common/periodic_task.py:171
DEBUG sahara.service.periodic Updating job statuses from (pid=17022) update_job_statuses /home/ubuntu/sahara/sahara/sahara/service/periodic.py:62
DEBUG sahara.openstack.common.loopingcall Dynamic looping call sleeping for 44.92 seconds from (pid=17022) _inner /home/ubuntu/sahara/sahara/sahara/openstack/common/loopingcall.py:130
DEBUG sahara.utils.ssh_remote [testcluster-master-001] _execute_command took 19.1 seconds to complete from (pid=17022) _log_command /home/ubuntu/sahara/sahara/sahara/utils/ssh_remote.py:407
DEBUG sahara.plugins.vanilla.v2_3_0.run_scripts Datanode count='' from (pid=17022) _check_datanodes_count /home/ubuntu/sahara/sahara/sahara/plugins/vanilla/v2_3_0/run_scripts.py:153
ERROR sahara.service.api Can't start services for cluster 'testcluster' (reason: invalid literal for int() with base 10: '')
TRACE sahara.service.api Traceback (most recent call last):
TRACE sahara.service.api File "/home/ubuntu/sahara/sahara/sahara/service/api.py", line 220, in _provision_cluster
TRACE sahara.service.api plugin.start_cluster(cluster)
TRACE sahara.service.api File "/home/ubuntu/sahara/sahara/sahara/plugins/vanilla/plugin.py", line 58, in start_cluster
TRACE sahara.service.api cluster.hadoop_version).start_cluster(cluster)
TRACE sahara.service.api File "/home/ubuntu/sahara/sahara/sahara/plugins/vanilla/v2_3_0/versionhandler.py", line 67, in start_cluster
TRACE sahara.service.api run.await_datanodes(cluster)
TRACE sahara.service.api File "/home/ubuntu/sahara/sahara/sahara/plugins/vanilla/v2_3_0/run_scripts.py", line 129, in await_datanodes
TRACE sahara.service.api if _check_datanodes_count(r, datanodes_count):
TRACE sahara.service.api File "/home/ubuntu/sahara/sahara/sahara/plugins/vanilla/v2_3_0/run_scripts.py", line 155, in _check_datanodes_count
TRACE sahara.service.api return exit_code == 0 and int(stdout) == count
TRACE sahara.service.api ValueError: invalid literal for int() with base 10: ''

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.1.mira2.git.0c123e1.b0661a2

Changeset: https://review.fuel-infra.org/1336
project: openstack/sahara
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara-2014.1.3-fuel5.1.1.mira2.git.0c123e1.b0661a2.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-1336/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.1~mira2+git.0c123e1.b0661a2

Changeset: https://review.fuel-infra.org/1336
project: openstack/sahara
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara_2014.1.3-fuel5.1.1~mira2+git.0c123e1.b0661a2_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-1336/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2.mira2.git.ba15daa.b0661a2

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara-2014.1.3-fuel5.1.2.mira2.git.ba15daa.b0661a2.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-1337/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2~mira2+git.ba15daa.b0661a2

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara_2014.1.3-fuel5.1.2~mira2+git.ba15daa.b0661a2_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-1337/ubuntu

Dmitry Mescheryakov (dmitrymex)
Changed in mos:
importance: Undecided → Critical
Dmitry Mescheryakov (dmitrymex)
Changed in mos:
assignee: Alexander Ignatov (aignatov) → Dmitry Mescheryakov (dmitrymex)
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2.mira3.git.cabce3b.b0661a2

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Dmitry Mescheryakov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara-2014.1.3-fuel5.1.2.mira3.git.cabce3b.b0661a2.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-1337/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2~mira3+git.cabce3b.b0661a2

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Dmitry Mescheryakov
subject: Fixed a ValueError on provisioning cluster
status: patchset-created

Files placed on repository:
sahara_2014.1.3-fuel5.1.2~mira3+git.cabce3b.b0661a2_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-1337/ubuntu

Dmitry Mescheryakov (dmitrymex)
Changed in mos:
assignee: Dmitry Mescheryakov (dmitrymex) → Alexander Ignatov (aignatov)
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2.mira3

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Dmitry Mescheryakov
subject: Fixed a ValueError on provisioning cluster
status: change-merged

Files placed on repository:
sahara-2014.1.3-fuel5.1.2.mira3.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package sahara has been built for project openstack/sahara
Package version == 2014.1.3, package release == fuel5.1.2~mira3

Changeset: https://review.fuel-infra.org/1337
project: openstack/sahara
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Dmitry Mescheryakov
subject: Fixed a ValueError on provisioning cluster
status: change-merged

Files placed on repository:
sahara_2014.1.3-fuel5.1.2~mira3_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

Alexander Ignatov (aignatov)
Changed in mos:
status: Confirmed → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.