404 from Heat when creating cluster from Quickstart doc

[Jason Dunsmore]

If I create a new Devstack server and run the following commands from the Quickstart document (http://docs.openstack.org/developer/magnum/dev/quickstart.html):

test -f ~/.ssh/id_rsa.pub || ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
nova keypair-add --pub-key ~/.ssh/id_rsa.pub testkey
magnum cluster-template-create --name k8s-cluster-template \
                       --image fedora-atomic-latest \
                       --keypair testkey \
                       --external-network public \
                       --dns-nameserver 8.8.8.8 \
                       --flavor m1.small \
                       --docker-volume-size 5 \
                       --network-driver flannel \
                       --coe kubernetes
magnum cluster-create --name k8s-cluster \
                      --cluster-template k8s-cluster-template \
                      --node-count 1

The cluster-create fails with reason "Stack with id 9df444d5-851c-4bd6-97e7-1993f1ab4921 not found in Heat." The following messages can be seen in the magnum-conductor output:

2016-12-19 21:16:16.750 26034 DEBUG heatclient.common.http [-] curl -g -i -X GET -H 'X-Auth-Token: xxx' -H 'Content-Type: application/json' -H 'X-Auth-Url: http://1.2.3.4:5000/v3' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://1.2.3.4:8004/v1/dcea1c8d8f2a4fd4923646e25f958d35/stacks/9df444d5-851c-4bd6-97e7-1993f1ab4921 log_curl_request /usr/local/lib/python2.7/dist-packages/heatclient/common/http.py:142
2016-12-19 21:16:16.819 26034 DEBUG heatclient.common.http [-]
HTTP/1.1 404 Not Found
Content-Type: application/json; charset=UTF-8
Content-Length: 618
X-Openstack-Request-Id: req-8cd134c9-5dbb-42d8-a755-2bfd27a68758
Date: Mon, 19 Dec 2016 21:16:16 GMT
Connection: keep-alive

{"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Stack (9df444d5-851c-4bd6-97e7-1993f1ab4921) could not be found.", "traceback": "Traceback (most recent call last):\n\n File \"/opt/stack/heat/heat/common/context.py\", line 424, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/opt/stack/heat/heat/engine/service.py\", line 478, in identify_stack\n raise exception.EntityNotFound(entity='Stack', name=stack_name)\n\nEntityNotFound: The Stack (9df444d5-851c-4bd6-97e7-1993f1ab4921) could not be found.\n", "type": "EntityNotFound"}, "title": "Not Found"}

If I do an "openstack stack list", I see stack "9df444d5-851c-4bd6-97e7-1993f1ab4921" exists in CREATE_COMPLETE status.

[Jason Dunsmore (jasondunsmore)]

elder magnum[master] $ git bisect start
elder magnum[master] $ git bisect good f2f9bd4653f
elder magnum[master] $ git bisect bad 2f9f613c9ac3e0
Bisecting: 34 revisions left to test after this (roughly 5 steps)
[5ede934aafa1629149168af376f8ada1ae2d7145] Merge "Disable horizon, swift and ceilometer"
elder magnum[(no branch, bisect started on master)] $ git bisect good
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[43dabfff37edb0ef2249e70462aede9b585c4fae] Merge "Consolidate heat network resources"
elder magnum[(no branch, bisect started on master)] $ git bisect bad Bisecting: 8 revisions left to test after this (roughly 3 steps)
[cfce4315814ce79f5fa6d2907a0e1df6e3ec19cc] Merge "Change gate Fedora Atomic image to the automated f24 build"
elder magnum[(no branch, bisect started on master)] $ git bisect good
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[0f4bd421f1974356bbaa43f03b3c68c5ed7b5b4e] Merge "Move cluster status updates into driver"
elder magnum[(no branch, bisect started on master)] $ git bisect bad
Bisecting: 2 revisions left to test after this (roughly 1 step)
[da04fe8b1c28c139ca85219edd4b78bcc6d9d11e] Merge "[install] Fix endpoint creation"
elder magnum[(no branch, bisect started on master)] $ git bisect good
Bisecting: 1 revision left to test after this (roughly 1 step)
[759c1b3b2b46b6af6688e5d4856de2c4ad2ef499] Move cluster status updates into driver
elder magnum[(no branch, bisect started on master)] $ git bisect bad
759c1b3b2b46b6af6688e5d4856de2c4ad2ef499 is the first bad commit
commit 759c1b3b2b46b6af6688e5d4856de2c4ad2ef499
Author: Randall Burt <email address hidden>
Date: Thu Dec 1 19:32:42 2016 -0600

    Move cluster status updates into driver

    This is an alternative implementation to:

    https://review.openstack.org/#/c/397961

    This version implements an earlier proposal from the
    spec that adds a driver method for synchronizing
    cluster state. This method is optional so that drivers
    that do not wish to leverage the existing periodic
    synchronization task can do so in whatever manner
    they wish and Magnum will not force them to do anything
    unnecessarily.

    1. add an update_cluster_status method to the driver
       interface
    2. implment update_cluster_status for Heat drivers
       using the existing tested logic
    3. Remove cluster status updates from the cluster conductor
       in favor of the periodic sync_cluster_status task - this
       should avoid timeouts and race conditions possible in the
       previous implementation
    4. Update the periodic sync_cluster_status method to use
       the driver to update cluster status rather than calling
       Heat directly

    Change-Id: Iae0ec7af2542343cc51e85f0efd21086d693e540
    Partial-Blueprint: bp-driver-consolodation

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/410824
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=c40413518af8ae4f7415140e6006ec50653024a1
Submitter: Jenkins
Branch: master

commit c40413518af8ae4f7415140e6006ec50653024a1
Author: Randall Burt <email address hidden>
Date: Wed Dec 14 09:47:43 2016 -0600

    Use correct context synching status

    Use the trust context when synching cluster status
    with orchestration status.

    Change-Id: I8ae0d1b92c3adce83032bb6c5f269d8d23c20c5e
    Partial-Blueprint: bp-driver-consolodation
    Closes-Bug: #1651243

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 4.0.0 release.