HTTPError: 400 Client Error: Bad Request for url

[Kevin]

Deploying

Used fuel 9.0 to deploy Openstack mitaka with Sahara.

http://sahara-files.mirantis.com/images/upstream/mitaka/
Downloaded: sahara-mitaka-ambari-2.2-centos-7.qcow2

Went through the cluster creation guide.

Selected HDP Plugin: HDP Plugin 2.3 (there is no image with 2.2 and no plugin with 2.2)
Created Master Node Group Template with basically bare minimum settings

Ambari
NameNode
JournalNode
ZooKeeper
Hive Metastore
HiveServer
YARN Timeline Server
MapReduce History Server
ResourceManager

Created slave with bare minimum

DataNode
NodeManager

I don't know the username and password to Ambari (I assume the deployment changed the default username and password). But Ambari does appear to be up and running.

Deployment fails with:

2016-06-24 22:37:34.359 20532 DEBUG sahara.utils.ssh_remote [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Created standard HTTP session for 192.168.0.239:8080 _get_http_client /usr/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:252
2016-06-24 22:37:34.361 20532 DEBUG sahara.utils.ssh_remote [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Caching session <requests.sessions.Session object at 0x7fa512b3f910> for 192.168.0.239:8080 _get_http_client /usr/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:260
2016-06-24 22:37:34.422 20532 DEBUG sahara.utils.ssh_remote [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: 1875d1d1-f886-4f9f-a452-e3d4778c6eab, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Closing HTTP session for 192.168.0.239:8080 _log_command /usr/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:798
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Error during operating on cluster (reason: 400 Client Error: Bad Request for url: http://192.168.0.239:8080/api/v1/blueprints/TestCluster)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Traceback (most recent call last):
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/service/ops.py", line 192, in wrapper
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] f(cluster_id, *args, **kwds)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/service/ops.py", line 293, in _provision_cluster
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] plugin.configure_cluster(cluster)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/plugins/ambari/plugin.py", line 87, in configure_cluster
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] deploy.create_blueprint(cluster)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/plugins/ambari/deploy.py", line 246, in create_blueprint
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] return client.create_blueprint(cluster.name, bp)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/plugins/ambari/client.py", line 119, in create_blueprint
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] return self.check_response(resp)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/sahara/plugins/ambari/client.py", line 78, in check_response
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] resp.raise_for_status()
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] File "/usr/lib/python2.7/dist-packages/requests/models.py", line 840, in raise_for_status
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] raise HTTPError(http_error_msg, response=self)
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] HTTPError: 400 Client Error: Bad Request for url: http://192.168.0.239:8080/api/v1/blueprints/TestCluster
2016-06-24 22:37:34.595 20532 ERROR sahara.service.ops [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3]
2016-06-24 22:37:35.108 20532 DEBUG keystoneclient.auth.identity.v3.base [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Making authentication request to http://192.168.10.4:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:188
2016-06-24 22:37:36.451 20532 INFO sahara.utils.cluster [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Cluster status has been changed. New status=Error
2016-06-24 22:37:36.452 20532 DEBUG sahara.utils.notification.sender [req-9fa92ba6-9ee2-4b4c-9ab4-9112d8dc9347 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: none, cluster: fdcac3f8-cf15-4592-9e63-129237718ab3] Notification about cluster is going to be sent. Notification type=sahara.cluster.update _notify /usr/lib/python2.7/dist-packages/sahara/utils/notification/sender.py:63
2016-06-24 22:37:38.436 20532 DEBUG oslo_service.periodic_task [-] Running periodic task SaharaPeriodicTasks.update_job_statuses run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215

[Vitalii Gridnev (vgridnev)]

About username / password: it should be displayed at UI in cluster details page after deployment.

About issue: at first glance it's not clear what can be a reason of this issue, I will try to investigate. Current idea: most probably it can be issue of validation in sahara, and probably cluster can't be deployed without Oozie Server (Ambari thinks that such situation is invalid), but need to check that.

[Vitalii Gridnev (vgridnev)]

About image: it's absolutely correct to use this image, just because it includes Ambari 2.2; but using this Ambari 2.2 we are deploying HDP stack 2.3.

[Vitalii Gridnev (vgridnev)]

Or without SecondaryNameNode, not sure.

[Vitalii Gridnev (vgridnev)]

Could you please try the following templates which we are using for testing ambari plugin right now:

https://github.com/openstack/sahara-tests/blob/master/sahara_tests/scenario/defaults/ambari-2.3.yaml.mako#L9-L30

[Vitalii Gridnev (vgridnev)]

It's validation issue. Cluster can't be created without secondarynamenode.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/334188

[Luigi Toscano (ltoscano)]

Out of curiosity, is this an HDP 2.3 requirement?

[Vitalii Gridnev (vgridnev)]

Luigi, seems like yes, because something like that we be raised:
Cluster Topology validation failed. Invalid service component count: [SECONDARY_NAMENODE(actual=0, required=1).

Also for CDH plugin, one secondary namenode is also a requirement.

[0] https://community.hortonworks.com/questions/38996/ambari-blueprint-validation-fails-for-a-namenode-h.html

[Kevin (kvasko)]

Adding "SecondaryName Node" to my template allows it to progress past this error.

I am having timeout (I think its because the internet is so slow) when the deployment is being performed. Is it possible to clone the HDP repositories and have the deployment use a local mirror?

[Vitalii Gridnev (vgridnev)]

Sorry for late answer, you can download HDP stack repos from [0]. Then you need to setup Apache so that other nodes will pick up that by HTTP server. When it's done, you can put direct urls to HDP and HDP utils repos in general configs section of cluster template.

Also, you can build image with ready mirror by saharaimagebuilder. to do so, you need to make few manual code changes to code of sahara-image-elements. I recommending to do so with plain image type. At line [1] you can export url variables from readme [2], and 'hdp-local-mirror' should be included into elements sequence at [3].

Please, note that HDP mirros are quite huge (final image is around 6GB or larger). Do not forget for DIB_MIN_TMPFS when running image building.

Additionally, I will try to make process of image building with mirror more automate soon.

[3] https://github.com/openstack/sahara-image-elements/blob/master/diskimage-create/diskimage-create.sh#L822
[2] https://github.com/openstack/sahara-image-elements/blob/master/elements/hdp-local-mirror/README.rst
[1] https://github.com/openstack/sahara-image-elements/blob/master/diskimage-create/diskimage-create.sh#L839
[0] http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.0/bk_Installing_HDP_AMB/content/_hdp_23_repositories.html

[Kevin (kvasko)]

Thanks! I'm trying this right now. I can try working on adding this if you provide some guidance on how you would like it implemented.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/334188
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=1bb6c999a19e582af3e088fcab98159fc7dfd18d
Submitter: Jenkins
Branch: master

commit 1bb6c999a19e582af3e088fcab98159fc7dfd18d
Author: Vitaly Gridnev <email address hidden>
Date: Sat Jun 25 19:05:12 2016 +0300

    forbid cluster creation without secondarynamenode

    if ambari blueprint was submitted without secondary namenode, we will
    recieve ValidationError. This affects only clusters without enabled HA
    mode.

    Change-Id: I88d429b9b5ba509dc6f3983f442659e0ec8af141
    Closes-bug: 1596113

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/sahara 5.0.0.0b2 development milestone.