Validation SSH keys are not copied to the overcloud nodes

[Tomas Sedovic]

After `openstack overcloud deploy ...` finishes, it should be possible to run the validations from the UI or via Mistral.

This is not the case though and instead, the (overcloud) validations error out with: "Failed to connect to the host via ssh."

As far as I could discover so far, it's because the deploy_ssh_keys workflow gets an empty list of servers to copy the SSH keys to. The workflow that sets up the keys is here:

https://github.com/openstack/tripleo-common/blob/8aa75683cbf796f8c0dc0334c03ef5023c96ca72/workbooks/deployment.yaml#L121

which invokes copy_ssh_key:

https://github.com/openstack/tripleo-common/blob/8aa75683cbf796f8c0dc0334c03ef5023c96ca72/workbooks/validations.yaml#L183

When I run `mistral task-list`, there are no errors there, but the `get_servers` task returns an empty list (even though this should be a complete deployment now and all the nodes should be in nova).

Running `mistral run-action nova.servers_list` does return the expected result.

And running `mistral execution-create tripleo.validations.v1.copy_ssh_key` manually after the deployment does get the list of servers and uploads the keys properly.

[Tomas Sedovic (tsedovic)]

The Heat deployment task is asynchronous and `copy_ssh_key` is therefore run as soon as we ask Heat to create the stack. I.e. before the servers even exist in Nova.

I think the keys should be set up by Heat during stack creation itself though, not separately like they are now.

We may be able to utilise the tripleo deploy artifacts here instead of modifying tripleo-heat-templates:

http://hardysteven.blogspot.cz/2016/08/tripleo-deploy-artifacts-and-puppet.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/391093

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/392971

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jon Schlueter (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/392971
Reason: going to abandon till master patch get's landed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390854
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b4bfc17728c7b1d507516d2f2ce24e2a2cc33efd
Submitter: Jenkins
Branch: master

commit b4bfc17728c7b1d507516d2f2ce24e2a2cc33efd
Author: Steven Hardy <email address hidden>
Date: Wed Oct 26 13:15:06 2016 +0100

    Add an optional extra node admin ssh key parameter

    This can be used to pass the e.g. the tripleo-validations ssh key into
    the deployment.

    Change-Id: I861b9e2252a9c8122dcf7df261386f1ea5200c4f
    Related-Bug: #1635226

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/394446

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/394446
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=67132afe98217475b733a3e44901c02883a52d87
Submitter: Jenkins
Branch: stable/newton

commit 67132afe98217475b733a3e44901c02883a52d87
Author: Steven Hardy <email address hidden>
Date: Wed Oct 26 13:15:06 2016 +0100

    Add an optional extra node admin ssh key parameter

    This can be used to pass the e.g. the tripleo-validations ssh key into
    the deployment.

    Change-Id: I861b9e2252a9c8122dcf7df261386f1ea5200c4f
    Related-Bug: #1635226
    (cherry picked from commit b4bfc17728c7b1d507516d2f2ce24e2a2cc33efd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391093
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=65b276d3ec744c85c3b70b40f417cb02ea38776d
Submitter: Jenkins
Branch: master

commit 65b276d3ec744c85c3b70b40f417cb02ea38776d
Author: Tomas Sedovic <email address hidden>
Date: Thu Oct 27 12:10:18 2016 +0200

    Fix the validation ssh keys workflow

    The current workflow did not end up copying the validation keys to the
    overcloud nodes, because the overcloud deployment action is
    asynchronous. So copy_ssh_keys runs right after the Heat call when the
    list of Nova servers is empty.

    So instead, we optionally pass the validation key into Heat directly
    at deployment time, not after.

    Change-Id: I5ceee2a1db077167253f7b3b7d6160f1efd0e447
    Closes-Bug: #1635226
    Depends-On: I861b9e2252a9c8122dcf7df261386f1ea5200c4f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/392971
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=63ead0f834fc8897d97fa5667af9ad8d5565312d
Submitter: Jenkins
Branch: stable/newton

commit 63ead0f834fc8897d97fa5667af9ad8d5565312d
Author: Tomas Sedovic <email address hidden>
Date: Thu Oct 27 12:10:18 2016 +0200

    Fix the validation ssh keys workflow

    The current workflow did not end up copying the validation keys to the
    overcloud nodes, because the overcloud deployment action is
    asynchronous. So copy_ssh_keys runs right after the Heat call when the
    list of Nova servers is empty.

    So instead, we optionally pass the validation key into Heat directly
    at deployment time, not after.

    Change-Id: I5ceee2a1db077167253f7b3b7d6160f1efd0e447
    Closes-Bug: #1635226
    Depends-On: I861b9e2252a9c8122dcf7df261386f1ea5200c4f
    (cherry picked from commit 65b276d3ec744c85c3b70b40f417cb02ea38776d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 5.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 5.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 5.4.0 release.