Creating cluster failed: External network not reachable

[Jai]

I am using the Quickstart guide available at https://docs.openstack.org/sahara/ocata/devref/quickstart.html
Getting the following ERROR. Please help...

Name my-cluster-1
Project ID 78ab9da540314b23906d0730e24b3807
ID 99185ea7-a06c-4b66-8199-5e11256877b4
Description None
Status Error
Status description
    Creating cluster failed for the following reason(s): Heat stack failed with status Resource CREATE failed: NotFound: resources.vanilla-default-master.resources[0].resources.floating_ip: External network babbdae3-a3b4-4625-b916-1e4484a40fe3 is not reachable from subnet 9fcc4498-101f-412f-9845-d2061e86153a. Therefore, cannot associate Port ef529cf7-31bb-4b1c-a5ae-abc994c97cf9 with a Floating IP. Neutron server returns request_ids: ['req-a10aaa0a-9065-4bbb-a6b8-eca62fe6142a'] Error ID: b97935b3-19d0-4988-97e5-5ef8fc9d97e2, Cannot delete heat stack my-cluster-199185ea7, reason: stack status: FAILED, status reason: Resource CREATE failed: NotFound: resources.vanilla-default-master.resources[0].resources.floating_ip: External network babbdae3-a3b4-4625-b916-1e4484a40fe3 is not reachable from subnet 9fcc4498-101f-412f-9845-d2061e86153a. Therefore, cannot associate Port ef529cf7-31bb-4b1c-a5ae-abc994c97cf9 with a Floating IP. Neutron server returns request_ids: ['req-a10aaa0a-9065-4bbb-a6b8-eca62fe6142a'] Error ID: c8c30b03-f836-4329-8553-97d6fc15de9d

Plugin vanilla
Version 2.7.1
Template vanilla-default-cluster
Base Image sahara-vanilla-latest-ubuntu
Neutron Management Network public
Keypair mystack
Use auto-configuration False
Public no
Protected no
Anti-affinity enabled for no processes

[Luigi Toscano (ltoscano)]

Can you please share more details about the deployment? This looks like a configuration problem. The error says:

External network babbdae3-a3b4-4625-b916-1e4484a40fe3 is not reachable from subnet 9fcc4498-101f-412f-9845-d2061e86153a. Therefore, cannot associate Port ef529cf7-31bb-4b1c-a5ae-abc994c97cf9 with a Floating IP.

What is network babbdae3-a3b4-4625-b916-1e4484a40fe3 ? And subnet 9fcc4498-101f-412f-9845-d2061e86153a? Are they connected?

[Jai (jaiparimi)]

Thanks for the reply.

Configuration: devStack - AIO on Ubuntu1604 VM - Openstqack Queen's

They are connected. Output from the command 'openstack network list'

+--------------------------------------+-------------+----------------------------------------------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-------------+----------------------------------------------------------------------------+
| 43f7d5a3-8847-4b65-9bfa-04e9afe5d975 | admin_net | 900e56df-1773-4cdc-98df-5518cfb4e73d |
| abaeae8d-ffef-4ff4-a9bb-eb54df2d7eef | selfservice | |
| babbdae3-a3b4-4625-b916-1e4484a40fe3 | public | 15e8068c-5104-4f54-bf5d-005b1f45db63, 9fcc4498-101f-412f-9845-d2061e86153a |
| c78ad2d9-f944-4777-b756-ab7675d62685 | private | 14480ea0-8481-4560-87eb-813c4ba2cefa, dc920868-328f-4acb-98f7-19f2a7bca75c |
| ddc9202d-b997-4819-a211-0f9ee51ab418 | alt-private | 1316c346-1b9d-49e2-b5b1-b86b4ca61796, d1c40b11-efd8-4816-8767-dad4bb14558e |
| fd222274-75ba-4871-a3b5-8e8133b3a3fc | trove-mgmt | a819b83a-d977-4d92-b016-e52022454114, b8627f10-5988-4a68-b0ca-3e68c021a7c0 |
+--------------------------------------+-------------+----------------------------------------------------------------------------+

I haven't changed any parameters and am following the guide from openstack.
Steps followed:

1. Built an image: tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i ubuntu -u

2. Uploaded the image: openstack image create sahara-vanilla-latest-ubuntu --disk-format qcow2 --container-format bare --file /opt/stack/sahara-image-elements/ubuntu_sahara_vanilla_hadoop_2_7_1_latest.qcow2

3. Registered the image: openstack dataprocessing image register sahara-vanilla-latest-ubuntu --username ubuntu tags add sahara-vanilla-latest-ubuntu --tags vanilla 2.7.1

4. Create Node Group template:
4A. openstack dataprocessing node group template create --name vanilla-default-master --plugin vanilla --plugin-version 2.7.1 --processes namenode resourcemanager --flavor 2 --auto-security-group --floating-ip-pool babbdae3-a3b4-4625-b916-1e4484a40fe3
4B. openstack dataprocessing node group template create --name vanilla-default-worker --plugin vanilla --plugin-version 2.7.1 --processes datanode nodemanager --flavor 2 --auto-security-group --floating-ip-pool babbdae3-a3b4-4625-b916-1e4484a40fe3

5. Create Cluster Template: openstack dataprocessing cluster template create --name vanilla-default-cluster --node-groups vanilla-default-master:1 vanilla-default-worker:3

6. Launch Cluster: openstack dataprocessing cluster create --name my-cluster-1 --cluster-template vanilla-default-cluster --user-keypair mystack --neutron-network public --image sahara-vanilla-latest-ubuntu wait

7. Spawning and then Error.

[Luigi Toscano (ltoscano)]

--neutron-network public - -> should be "private" (as in the guide).

[Jai (jaiparimi)]

Sorry, I tried with private first and then changed it to public and tried again

The error when I used private:

Creating cluster failed for the following reason(s): Heat stack failed with status Resource CREATE failed: BadRequest: resources.vanilla-default-master.resources[0].resources.floating_ip: Bad floatingip request: Network c78ad2d9-f944-4777-b756-ab7675d62685 is not a valid external network. Neutron server returns request_ids: ['req-4f0e4aca-3fef-4bb6-aecc-5f607a280361'] Error ID: 8eda2064-b9af-4913-bdbd-ecfec5349dd2, Cannot delete heat stack my-cluster-170c00ef0, reason: stack status: FAILED, status reason: Resource CREATE failed: BadRequest: resources.vanilla-default-master.resources[0].resources.floating_ip: Bad floatingip request: Network c78ad2d9-f944-4777-b756-ab7675d62685 is not a valid external network. Neutron server returns request_ids: ['req-4f0e4aca-3fef-4bb6-aecc-5f607a280361'] Error ID: 916116a6-a691-4bc9-9a87-8e6221fd672b

[Luigi Toscano (ltoscano)]

Just to be sure, where did you try private? There are two different network configuration:
- the floating ip pool in the node group templates is on the public network
- the neutron network in the cluster is the private network

[Jai (jaiparimi)]

When I tried private:
  Node Group Template: private pool id c78ad2d9-f944-4777-b756-ab7675d62685 in floating ip pool
  Cluster: private in neutron network
When I tried the public:
  Node Group Template: public pool id babbdae3-a3b4-4625-b916-1e4484a40fe3 in floating ip pool
  Cluster: public in neutron network

[Luigi Toscano (ltoscano)]

The documentation says:
- node group template: public pool id
- cluster: private

[Jai (jaiparimi)]

I tried launching with public pool id in node group template and private in cluster launch.
The cluster status is stuck in spawning state for more than an hour.

[Jai (jaiparimi)]

After 1h30m, Creating cluster failed for the following reason(s): Heat stack failed with status Resource CREATE failed: WaitConditionTimeout: resources.vanilla-default-master.resources[0].resources.vanilla-default-master-wc-waiter: 0 of 1 received Error ID: 46957a5f-2ab2-4e82-86f1-a0b9f264d870

Name my-cluster-1
Project ID 7a92acaf7e154c159f580ab3e95b0325
ID 87b178c9-333f-4022-93ea-048d7d7bcebb
Description None
Status Error
Status description
    Creating cluster failed for the following reason(s): Heat stack failed with status Resource CREATE failed: WaitConditionTimeout: resources.vanilla-default-master.resources[0].resources.vanilla-default-master-wc-waiter: 0 of 1 received Error ID: 46957a5f-2ab2-4e82-86f1-a0b9f264d870

Plugin vanilla
Version 2.7.1
Template vanilla-default-cluster
Base Image sahara-vanilla-latest-ubuntu
Neutron Management Network private
Keypair mystack
Use auto-configuration False
Public no
Protected no
Anti-affinity enabled for no processes

Name: vanilla-default-master
Number of Nodes 0
Flavor m1.small
Floating IP Pool public
Template vanilla-default-master
Use auto-configuration False
Proxy Gateway no
Auto Security Group yes
Security Groups
Node Processes namenode, resourcemanager
HDFS placement Drive type Ephemeral drive

[Luigi Toscano (ltoscano)]

Not sure why you marked the bug as "Invalid" while still investigating it, but:
- the original bug, as it is, it is not valid, as it was a configuration isuse;
- the issue that you are facing now most likely does not depend on Sahara, but on some resource problem.

It looks like Heat can't complete the provisioning of the requested instance because it can't reach them. Try to manually launch an instance using the private network, assign it a floating IP and, after opening the ICMP and/or SSH on the security group used, try to reach that instance through the floating IP.

[Jai (jaiparimi)]

Thanks again,
I changed the status to "invalid" because the original bug(in title) is invalid.
I will manually launch an instance as you suggested and get back to you with the results.

Meanwhile, I tried to launch the cluster today and it progressed further and failed in the "Starting" phase - Await Node start up. Details below

Info: 'Operation with name 'Executing "sudo su -lc "hdfs dfsadmin -report" hadoop | grep 'Live datanodes\|Datanodes available:' | grep -o '[0-9]\+' | head -n 1"'' timed out after 300 second(s) Error ID: d9bfe5f9-c3f4-45aa-9ebb-9a1cccd4e523

+---------------------------------------------------------------------------------+----------------------------------------------------+
| Field | Value |
+---------------------------------------------------------------------------------+----------------------------------------------------+
| Anti affinity | |
| Cluster template id | 8104aa49-3c03-44a4-bfb9-5fb11d4f2a4e |
| Description | None |
| Id | f535d561-4bfb-4f98-9833-b75830aacfb1 |
| Image | 3fec6fc2-a1f5-4736-bed5-74fae8b3747c |
| Info | {} |
| Is protected | False |
| Is public | False |
| Is transient | False |
| Name | my-cluster-1 |
| Neutron management network | e35e8ea8-b81f-4816-9e30-a2f72b48a4c0 |
| Node groups | vanilla-default-master:1, vanilla-default-worker:3 |
| Plugin name | vanilla |
| Plugin version | 2.7.1 |
| Status | Error |
| Use autoconfig | False |
| User keypair id ...