Followed this guide for bay creation using the same image. http://docs.openstack.org/developer/magnum/dev/dev-quickstart.html
Bay creation is stuck in "CREATE_IN_PROGRESS" state.
"kube_masters" heat resource is stuck in "CREATE_IN_PROGRESS" state. Although the nova instance creation is completed and I am able to login.
magnum-conductor logs -
=====================
ist-packages/routes/middleware.py\", line 136, in __call__\n response = self.app(environ, start_response)\n File \"/usr/local/lib/python2.7/dist-packages/webob/dec.py\", line 144, in __call__\n return resp(environ, start_response)\n File \"/usr/local/lib/python2.7/dist-packages/webob/dec.py\", line 130, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File \"/usr/local/lib/python2.7/dist-packages/webob/dec.py\", line 195, in call_func\n return self.func(req, *args, **kwargs)\n File \"/opt/stack/heat/heat/common/wsgi.py\", line 883, in __call__\n raise translate_exception(err, request.best_match_language())\nForbidden: You are not authorized to use global_index.\n", "type": "Forbidden"}, "title": "Forbidden"}
log_http_response /usr/local/lib/python2.7/dist-packages/heatclient/common/http.py:142
2015-09-24 10:43:47.919 4088 WARNING magnum.service.periodic [req-8436c811-7606-48b7-8088-4cf4d5771d19 - - - - -] Ignore error [ERROR: You are not authorized to use global_index.] when syncing up bay status.
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic Traceback (most recent call last):
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/home/ubuntu/magnum/magnum/service/periodic.py", line 69, in sync_bay_status
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic sid_to_stack_mapping = {s.id: s for s in stacks}
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/home/ubuntu/magnum/magnum/service/periodic.py", line 69, in <dictcomp>
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic sid_to_stack_mapping = {s.id: s for s in stacks}
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 101, in paginate
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic stacks = self._list(url, 'stacks')
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/openstack/common/apiclient/base.py", line 131, in _list
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic body = self.client.get(url).json()
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 284, in get
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic return self.client_request("GET", url, **kwargs)
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 277, in client_request
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic resp, body = self.json_request(method, url, **kwargs)
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic resp = self._http_request(url, method, **kwargs)
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 221, in _http_request
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic raise exc.from_response(resp)
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic HTTPForbidden: ERROR: You are not authorized to use global_index.
2015-09-24 10:43:47.919 4088 ERROR magnum.service.periodic
Heat engine logs -
==================
2015-09-24 10:45:26.212 DEBUG heat.engine.scheduler [-] Task Stack "k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e" [cde190ac-60ad-4974-97e2-45659be6a9d8] Update running from (pid=3837) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.212 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e" [cde190ac-60ad-4974-97e2-45659be6a9d8] Update running from (pid=3837) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.250 INFO heat.engine.environment [-] Registering file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubeminion.yaml -> file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubeminion.yaml
2015-09-24 10:45:26.252 DEBUG heat.engine.scheduler [-] Task update_task from Stack "k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e" [cde190ac-60ad-4974-97e2-45659be6a9d8] sleeping from (pid=3837) _sleep /opt/stack/heat/heat/engine/scheduler.py:167
2015-09-24 10:45:26.419 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "k8sbay-w5jvb4xmu3y7" [3edc2d1c-0058-41a8-8f06-ff11ca994f37] running from (pid=3834) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.419 DEBUG heat.engine.scheduler [-] Task resource_action running from (pid=3834) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.419 DEBUG heat.engine.scheduler [-] Task _run_to_completion from ResourceGroup "kube_masters" Stack "k8sbay-w5jvb4xmu3y7" [3edc2d1c-0058-41a8-8f06-ff11ca994f37] running from (pid=3834) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.462 INFO heat.engine.environment [-] Registering file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubeminion.yaml -> file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubeminion.yaml
2015-09-24 10:45:26.462 INFO heat.engine.environment [-] Registering file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubemaster.yaml -> file:///home/ubuntu/magnum/magnum/templates/heat-kubernetes/kubemaster.yaml
2015-09-24 10:45:26.464 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "k8sbay-w5jvb4xmu3y7" [3edc2d1c-0058-41a8-8f06-ff11ca994f37] sleeping from (pid=3834) _sleep /opt/stack/heat/heat/engine/scheduler.py:167
2015-09-24 10:45:26.915 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e-0-alanufipkty7" [4a198f65-3a4c-44fb-95f2-ed4a9942304d] running from (pid=3839) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.916 DEBUG heat.engine.scheduler [-] Task resource_action running from (pid=3839) step /opt/stack/heat/heat/engine/scheduler.py:223
2015-09-24 10:45:26.938 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e-0-alanufipkty7" [4a198f65-3a4c-44fb-95
cloud-init-log for master minion -
==================================
[minion@k8-pxsvag2j6e-0-alanufipkty7-kube-master-4qir2uvvb7gf ~]$ cat /var/log/cloud-init-output.log
Cloud-init v. 0.7.5 running 'init-local' at Thu, 24 Sep 2015 10:00:13 +0000. Up 85.81 seconds.
Cloud-init v. 0.7.5 running 'init' at Thu, 24 Sep 2015 10:00:28 +0000. Up 101.14 seconds.
ci-info: +++++++++++++++++++++++++Net device info+++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0: | True | 10.0.0.5 | 255.255.255.0 | fa:16:3e:90:de:2b |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
Cloud-init v. 0.7.5 running 'modules:config' at Thu, 24 Sep 2015 10:00:51 +0000. Up 123.66 seconds.
Cloud-init v. 0.7.5 running 'modules:final' at Thu, 24 Sep 2015 10:01:00 +0000. Up 133.36 seconds.
configuring kubernetes (master)
starting services
activating service etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
activating service kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
activating service kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
activating service kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
#!/bin/bash -v
curl -i -X POST -H 'X-Auth-Token: 9229798d2f8141d99e69ab299b7a31fe' -H 'Content-Type: application/json' -H 'Accept: application/json' http://192.168.6.2:8004/v1/22a181c772b643f7b53527fbbb70
2f4a/stacks/k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e-0-alanufipkty7/4a198f65-3a4c-44fb-95f2-ed4a9942304d/resources/master_wait_handle/signal --data-binary '{"status": "SUCCESS"}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 192.168.6.2 port 8004: No route to host
2015-09-24 10:01:24,696 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-010 [7]
Created symlink from /etc/systemd/system/multi-user.target.wants/flannel-config.service to /etc/systemd/system/flannel-config.service.
2015-09-24 10:01:27,515 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2015-09-24 10:01:27,636 - util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.py'>) fa
iled
Cloud-init v. 0.7.5 finished at Thu, 24 Sep 2015 10:01:30 +0000. Datasource DataSourceConfigDriveNet [net,ver=2][source=/dev/sr0]. Up 163.30 seconds
Contents of the file -
====================
[minion@k8-pxsvag2j6e-0-alanufipkty7-kube-master-4qir2uvvb7gf ~]$ sudo cat /var/lib/cloud/instance/scripts/part-010
#!/bin/bash -v
curl -i -X POST -H 'X-Auth-Token: 9229798d2f8141d99e69ab299b7a31fe' -H 'Content-Type: application/json' -H 'Accept: application/json' http://192.168.6.2:8004/v1/22a181c772b643f7b53527fbbb702f4a/stacks/k8sbay-w5jvb4xmu3y7-kube_masters-qrpxsvag2j6e-0-alanufipkty7/4a198f65-3a4c-44fb-95f2-ed4a9942304d/resources/master_wait_handle/signal --data-binary '{"status": "SUCCESS"}'
[minion@k8-pxsvag2j6e-0-alanufipkty7-kube-master-4qir2uvvb7gf ~]$
FYI, the same environment worked earlier. This issue exists even after rebooting and rejoining. All the other services are working fine.