swarn cluster creation WaitConditiontimeout

[manjunatha]

Hi
I using pike version of openstack and try to create a swarm cluster using magnum, but it failing after 1 hour 40 mintutes, initally it use timeout I have extended using timeout 180

[root@controller heat]# magnum cluster-show 0daa8c5f-3f3e-465e-b262-243274673904
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status | CREATE_FAILED |
| cluster_template_id | 29ec94f6-ca20-4d1b-bf22-d2e3af2274f8 |
| node_addresses | [] |
| docker_volume_size | - |
| stack_id | 3d5a8f25-5949-49d3-aacc-3e704f98fdc9 |
| status_reason | Resource CREATE failed: WaitConditionTimeout: resources.swarm_masters.resources[0].resources.master_wait_condition: 0 of 1 received |
| created_at | 2018-05-02T15:46:42+00:00 |
| name | swarm-cluster-010 |
| updated_at | 2018-05-02T17:30:05+00:00 |
| coe_version | 1.2.5 |
| faults | {'swarm_masters': 'WaitConditionTimeout: resources.swarm_masters.resources[0].resources.master_wait_condition: 0 of 1 received', 'master_wait_condition': 'WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received', '0': 'WaitConditionTimeout: resources[0].resources.master_wait_condition: 0 of 1 received'} |
| keypair | mykey |
| api_address | tcp://:2376 |
| master_addresses | ['10.5.24.61'] |
| create_timeout | 180 |
| node_count | 1 |
| discovery_url | https://discovery.etcd.io/0a25a9eac20525690659766cdddec9a3 |
| master_count | 1 |
| container_version | 1.12.6 |
| uuid | 0daa8c5f-3f3e-465e-b262-243274673904 |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@controller heat]#

And I found that heat-engine error

2018-05-02 16:48:34.138 2036 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating TemplateResource "api_address_switch" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:48:34.158 2038 INFO heat.engine.service [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] Creating stack swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73-api_address_switch-vvu7iwp4pf3g
2018-05-02 16:48:34.208 2038 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] convergence_dependencies: {}
2018-05-02 16:48:34.211 2038 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] [swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73-api_address_switch-vvu7iwp4pf3g(a3156da3-8f36-436c-86e1-41771642616a)] update traversal 17611131-6b5e-4a85-95c2-3a9d75841b4f complete
2018-05-02 16:48:34.212 2038 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] Stack CREATE COMPLETE (swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73-api_address_switch-vvu7iwp4pf3g): Stack CREATE completed successfully
2018-05-02 16:48:35.315 2035 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating Value "no_proxy_extended" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:48:35.487 2040 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating SoftwareConfig "write_swarm_master_service" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:48:35.507 2039 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating SoftwareConfig "write_heat_params" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:48:35.743 2042 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating MultipartMime "swarm_master_init" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:48:36.574 2041 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating Server "swarm-master" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:50:01.534 2038 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 16:50:01.542 2037 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] creating NoneResource "docker_volume_attach" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 18:30:01.774 2038 INFO heat.engine.resources.openstack.heat.wait_condition [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42] Timed out (0 of 1 received)
2018-05-02 18:30:01.779 2038 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42]
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource Traceback (most recent call last):
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 831, in _action_recorder
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource yield
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 939, in _do_action
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 351, in wrapper
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource step = next(subtask)
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 890, in action_handler_task
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource done = check(handler_data)
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource return self._wait(*data)
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource raise exc
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2018-05-02 18:30:01.779 2038 ERROR heat.engine.resource
2018-05-02 18:30:01.925 2038 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] Stack CREATE FAILED (swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf-0-jkl33wm7dn73): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2018-05-02 18:30:02.865 2036 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] CREATE: TemplateResource "0" [93b0668c-d8eb-4dc5-9cf6-91815a1e9c42] Stack "swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf" [9a55cdd2-d064-4bfc-9591-90c19a9ccc04]
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource Traceback (most recent call last):
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 831, in _action_recorder
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource yield
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 939, in _do_action
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 351, in wrapper
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource step = next(subtask)
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 890, in action_handler_task
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource done = check(handler_data)
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 370, in check_create_complete
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource return self._check_status_complete(self.CREATE)
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 420, in _check_status_complete
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource action=action)
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource ResourceFailure: WaitConditionTimeout: resources[0].resources.master_wait_condition: 0 of 1 received
2018-05-02 18:30:02.865 2036 ERROR heat.engine.resource
2018-05-02 18:30:02.881 2036 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] Stack CREATE FAILED (swarm-cluster-010-653snycy2fxk-swarm_masters-hmvafbdykvqf): Resource CREATE failed: WaitConditionTimeout: resources[0].resources.master_wait_condition: 0 of 1 received
2018-05-02 18:30:03.730 2037 INFO heat.engine.resource [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] CREATE: ResourceGroup "swarm_masters" [9a55cdd2-d064-4bfc-9591-90c19a9ccc04] Stack "swarm-cluster-010-653snycy2fxk" [3d5a8f25-5949-49d3-aacc-3e704f98fdc9]
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource Traceback (most recent call last):
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 831, in _action_recorder
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource yield
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 939, in _do_action
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 351, in wrapper
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource step = next(subtask)
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 890, in action_handler_task
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource done = check(handler_data)
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 375, in check_create_complete
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource return super(ResourceGroup, self).check_create_complete()
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 370, in check_create_complete
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource return self._check_status_complete(self.CREATE)
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 420, in _check_status_complete
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource action=action)
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource ResourceFailure: WaitConditionTimeout: resources.swarm_masters.resources[0].resources.master_wait_condition: 0 of 1 received
2018-05-02 18:30:03.730 2037 ERROR heat.engine.resource
2018-05-02 18:30:03.766 2037 INFO heat.engine.stack [req-86d4caa5-a892-4eda-97e0-c8f43bad4f38 - admin - default default] Stack CREATE FAILED (swarm-cluster-010-653snycy2fxk): Resource CREATE failed: WaitConditionTimeout: resources.swarm_masters.resources[0].resources.master_wait_condition: 0 of 1 received
[root@controller heat]#