kubernetes bay needs to be fully functional before notifying the wait condition and transitioning to CREATE_COMPLETE

[Corey O'Brien]

Currently a kubernetes bay goes to CREATE_COMPLETE before the kubernetes API is fully active. The template only waits for all the start commands to be issued but doesn't wait for all the kubernetes setup work to complete. As a result we get kubernetes failures in the gate due to the tests trying to connect to kubernetes before it is actually ready.

Example stack trace from test: http://paste.openstack.org/show/488773/
Example gate failure from bay that went CREATE_COMPLETE before kubernetes was functional:
http://logs.openstack.org/05/275405/6/check/gate-functional-dsvm-magnum-k8s/4c1c488/
 * The bay went to CREATE_COMPLETE at 05:12:36 (from m-cond.log)
 * master node was still creating and running podmaster and proxy and other startup containers as of 5:16:34, 5:17:23, etc.

[Corey O'Brien (coreypobrien)]

Magnum also needs to wait for minion nodes to be registered with the master

[Ton Ngo (ton-i)]

This also has implication when other tools interface with Magnum via the API. Gate is only one example.
We should ensure that when the bay status goes to CREATE_COMPLETE, the COE is fully functional.

[Corey O'Brien (coreypobrien)]

Agreed Ton. I want this issue to reflect the overall problem, not just the gate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275405
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=08ac9a6cbac5b324747c178c4cab36296c16b567
Submitter: Jenkins
Branch: master

commit 08ac9a6cbac5b324747c178c4cab36296c16b567
Author: Corey O'Brien <email address hidden>
Date: Tue Feb 2 15:57:40 2016 -0500

    Remove minion dependency on master

    Make thhe minions work correctly without depending on the master.
    Removing this dependency decreases provisioning time by provisioning
    nodes in parallel.

    Change-Id: I025dbf54ef0cdafc4fe26acc33a9faed8cacd1ca
    Partial-Bug: 1536739
    Partial-Bug: 1551824

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/301954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/301954
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=8972b0c1c4a45cf8fd766faeb0e3db0bcc51354c
Submitter: Jenkins
Branch: master

commit 8972b0c1c4a45cf8fd766faeb0e3db0bcc51354c
Author: Hongbin Lu <email address hidden>
Date: Tue Apr 5 18:18:06 2016 -0400

    Notify Heat only if kube-apiserver is running

    Kubernetes bay needs to be fully functional before notifying Heat
    to bring the stack to CREATE_COMPLETE. This patch added a systemd
    unit that keeps pinging the kube-apiserver and notify Heat once
    it goes through.

    Change-Id: Ib4fb22f3040d76d488c6cac6873d64b4d973ebb1
    Closes-Bug: #1551824

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 3.0.0 release.

[Spyros Trigazis (strigazi)]

I think we need to open this bug again, heat should be notified when scheduler and controller-manager are ready.

Thoughs?