| 2014-12-05 23:29:40 |
Andrew Lazarev |
description |
Steps to reproduce:
1. Create vanilla2 cluster. I used 1*["namenode", "resourcemanager", "historyserver", "oozie"] + 3*["datanode", "nodemanager"].
2. Scale cluster. I was trying to add 1*["datanode"].
Stacktrace:
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Traceback (most recent call last):
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/service/ops.py", line 141, in wrapper
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops f(cluster_id, *args, **kwds)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/service/ops.py", line 272, in _provision_scaled_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops plugin.scale_cluster(cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/plugin.py", line 60, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops cluster.hadoop_version).scale_cluster(cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/v2_4_1/versionhandler.py", line 106, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops sc.scale_cluster(self.pctx, cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/scaling.py", line 31, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops config.configure_instances(pctx, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 57, in configure_instances
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _provisioning_configs(pctx, instance)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 63, in _provisioning_configs
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _push_xml_configs(instance, xmls)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 261, in _push_xml_configs
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _push_configs_to_instance(instance, xml_confs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 268, in _push_configs_to_instance
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops r.write_file_to(fl, data, run_as_root=True)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 578, in write_file_to
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops self._run_s(_write_file_to, timeout, remote_file, data, run_as_root)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 643, in _run_s
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return self._run_with_log(func, timeout, *args, **kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 517, in _run_with_log
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return self._run(func, *args, **kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 640, in _run
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/procutils.py", line 53, in run_in_subprocess
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops SubprocessException: RemoteCommandException: Error during command execution: "mv temp-file-b3401ce5-2fa2-441b-ab39-3ecaa189a648 /opt/hive/conf/hive-site.xml"
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Return code: 1
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops STDERR:
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops mv: cannot move temp-file-b3401ce5-2fa2-441b-ab39-3ecaa189a648 to /opt/hive/conf/hive-site.xml: No such file or directory
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Error ID: 059ca13b-3d1a-414e-96fe-3cad80c80376
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.911 97815 INFO sahara.utils.general [-] Cluster status has been changed: id=93d765de-4bf4-4a89-ad3a-75c5d3b14888, New status=Error
Reproduced both on ubuntu and fedora images.
Note, stacktrace in master is hidden because of https://bugs.launchpad.net/sahara/+bug/1399490
It looks like problem was introduced by https://review.openstack.org/#/c/133186/ |
Steps to reproduce:
1. Create vanilla2 cluster. I used 1*["namenode", "resourcemanager", "historyserver", "oozie"] + 3*["datanode", "nodemanager"].
2. Scale cluster. I was trying to add 1*["datanode"].
Stacktrace:
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Traceback (most recent call last):
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/service/ops.py", line 141, in wrapper
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops f(cluster_id, *args, **kwds)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/service/ops.py", line 272, in _provision_scaled_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops plugin.scale_cluster(cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/plugin.py", line 60, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops cluster.hadoop_version).scale_cluster(cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/v2_4_1/versionhandler.py", line 106, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops sc.scale_cluster(self.pctx, cluster, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/scaling.py", line 31, in scale_cluster
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops config.configure_instances(pctx, instances)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 57, in configure_instances
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _provisioning_configs(pctx, instance)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 63, in _provisioning_configs
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _push_xml_configs(instance, xmls)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 261, in _push_xml_configs
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops _push_configs_to_instance(instance, xml_confs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/plugins/vanilla/hadoop2/config.py", line 268, in _push_configs_to_instance
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops r.write_file_to(fl, data, run_as_root=True)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 578, in write_file_to
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops self._run_s(_write_file_to, timeout, remote_file, data, run_as_root)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 643, in _run_s
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return self._run_with_log(func, timeout, *args, **kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 517, in _run_with_log
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return self._run(func, *args, **kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/ssh_remote.py", line 640, in _run
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops return procutils.run_in_subprocess(self.proc, func, args, kwargs)
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops File "/Users/alazarev/openstack/sahara/sahara/utils/procutils.py", line 53, in run_in_subprocess
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops SubprocessException: RemoteCommandException: Error during command execution: "mv temp-file-b3401ce5-2fa2-441b-ab39-3ecaa189a648 /opt/hive/conf/hive-site.xml"
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Return code: 1
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops STDERR:
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops mv: cannot move temp-file-b3401ce5-2fa2-441b-ab39-3ecaa189a648 to /opt/hive/conf/hive-site.xml: No such file or directory
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops Error ID: 059ca13b-3d1a-414e-96fe-3cad80c80376
2014-12-05 15:12:25.769 97815 TRACE sahara.service.ops
2014-12-05 15:12:25.911 97815 INFO sahara.utils.general [-] Cluster status has been changed: id=93d765de-4bf4-4a89-ad3a-75c5d3b14888, New status=Error
Reproduced both on ubuntu and fedora juno images.
Note, stacktrace in master is hidden because of https://bugs.launchpad.net/sahara/+bug/1399490
It looks like problem was introduced by https://review.openstack.org/#/c/133186/ |
|
