11.2.1: swift playbooks fails with missing swift_pubkey

[Bjoern]

git version 11.2.1-3-gfaeb18a

I'm getting this error while running os-swift-install after I did enable LDAP as keystone identity driver (don't think this is related).
The ansible inventory is a upgraded one from Juno

TASK: [os_swift_sync | Create authorized keys file from host vars] ************
fatal: [infra02_swift_proxy_container-9f9b0003] => One or more undefined variables: 'dict object' has no attribute 'swift_pubkey'
fatal: [swift-node2] => One or more undefined variables: 'dict object' has no attribute 'swift_pubkey'
fatal: [infra01_swift_proxy_container-32a66088] => One or more undefined variables: 'dict object' has no attribute 'swift_pubkey'
fatal: [infra03_swift_proxy_container-502e849a] => One or more undefined variables: 'dict object' has no attribute 'swift_pubkey'
fatal: [swift-node3] => One or more undefined variables: 'dict object' has no attribute 'swift_pubkey'

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/os-swift-install.retry

infra01_swift_proxy_container-32a66088 : ok=48 changed=6 unreachable=1 failed=0
infra02_swift_proxy_container-9f9b0003 : ok=48 changed=7 unreachable=1 failed=0
infra03_swift_proxy_container-502e849a : ok=48 changed=7 unreachable=1 failed=0
swift-node1 : ok=59 changed=3 unreachable=0 failed=1
swift-node2 : ok=70 changed=7 unreachable=1 failed=0
swift-node3 : ok=70 changed=10 unreachable=1 failed=0

[Bjoern (bjoern-t)]

Just an additional side note, the --limit parameter was not used

[Andy McCrae (andrew-mccrae)]

The key gets created as part of the os_swift role which is called as part of that play (or the os-swift-setup.yml play).

I notice there was an error on node1 before these failures (failed=1) what was that? Also can you confirm the os-swift-setup.yml play ran successfully first?

To clarify, the os-swift-isntall.yml calls 2 plays:

- os-swift-setup - sets up the swift bits (services, users etc)
- os-swift-sync: - syncs the ring/key files etc.

In order for the keys to be sync'd they need to have been created, so I'm interested to know whether the os-swift-setup play ran successfully first.

If that is the case, and the key does exist it would be useful to get a debug output from the keys themselves.

[Bjoern (bjoern-t)]

The error I do see is just rolling by when running the os-swift-install.yml. I have not looked up yet where this task is located

TASK: [os_swift | Ensure swift user] ******************************************
skipping: [swift-node3]
skipping: [swift-node2]
skipping: [infra01_swift_proxy_container-32a66088]
skipping: [infra03_swift_proxy_container-502e849a]
failed: [swift-node1] => {"attempts": 5, "failed": true, "parsed": false}
Task failed as maximum retries was encountered

[Andy McCrae (andrew-mccrae)]

node1 looks to be the issue - the swift-node1 fails out so it doesn't generate the swift_pubkey var for node1, and as a result the var doesn't exist for the other hosts to use (so they fail also).

The task that failed above is the task for adding the swift keystone user. I'm not sure why that would fail, but I believe that would be an issue with keystone itself rather than swift (its using the keystone module and has nothing to do with swift itself, except for that its the keystone user for swift).

Would it be possible to get a -vvv on that task or the os_swift play? It doesn't give too much insight into why the keystone swift user couldn't be created.

[Julian Montez (jpmontez)]

I'm receiving the same error. One task fails before the public keys are unable to synchronize, as well. It's the step right before Bjoern's "Ensure Swift user". In my case, it fails to "Ensure swift service".

I'm attaching the verbose output of my run-through in this comment.

[Jesse Pretorius (jesse-pretorius)]

We hit this issue and resolved it by using the slurp -> decode method instead of cat -> paste... see https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/c9773b9d9c85dbec0422839829a9dedbd07991d0 for details.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/266840

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266840
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=d954d599bdd888bb865b2674c2db2778571e9f43
Submitter: Jenkins
Branch: master

commit d954d599bdd888bb865b2674c2db2778571e9f43
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 13 11:16:00 2016 +0000

    Use slurp to collect the swift ssh keys

    Extracting the ssh public key using cat and storing the result in a
    fact has resulted in periodic failures in the collection of the key,
    and thereafter the failure to appropriately place that key into the
    authorised_keys file.

    This patch changes the collection method to use the Ansible slurp
    module which has been found to be more reliable.

    Closes-Bug: #1506291
    Change-Id: I3ba5cdc944f38c67762860dc0d06bd5c35eeb269

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/312785
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=c3f411d37a8e45c1d1da9a6b3ac1f63a5760c57a
Submitter: Jenkins
Branch: liberty

commit c3f411d37a8e45c1d1da9a6b3ac1f63a5760c57a
Author: Jesse Pretorius <email address hidden>
Date: Wed Jan 13 11:16:00 2016 +0000

    Use slurp to collect the swift ssh keys

    Extracting the ssh public key using cat and storing the result in a
    fact has resulted in periodic failures in the collection of the key,
    and thereafter the failure to appropriately place that key into the
    authorised_keys file.

    This patch changes the collection method to use the Ansible slurp
    module which has been found to be more reliable.

    Closes-Bug: #1506291
    Change-Id: I3ba5cdc944f38c67762860dc0d06bd5c35eeb269
    (cherry picked from commit d954d599bdd888bb865b2674c2db2778571e9f43)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 12.0.13 release.