Liberty to Mitaka upgrade fails inside deploy-config-changes.yml

[Bjoern]

Liberty to Mitaka upgrade fails inside deploy-config-changes.yml:

PLAY [Upgrade environment/inventory configuration] ****************************

TASK: [Create an old copy of openstack_deploy] ********************************
failed: [localhost] => {"checksum": "3c87aaa1594b90273fd89783a1f5dccbd5ba5f6d", "failed": true}
msg: Could not replace file: /root/.ansible/tmp/ansible-tmp-1476116920.84-129637924756635/source to /etc/openstack_deploy.LIBERTY/openstack_user_config.yml.haproxy: [Errno 20] Not a directory

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/deploy-config-changes.retry

localhost : ok=0 changed=0 unreachable=0 failed=1

ran 2/opt/rpc-openstack/openstack-ansible/scripts/upgrade-utilities/playbooks/deploy-config-changes.yml

This is based off 13.3.4-1-ga489036 and it can be fixed with removing /etc/openstack_deploy.LIBERTY/ (it was a faile openstack_user_config.yml) and replaced by a directory /etc/openstack_deploy.LIBERTY.

We should make sure that /etc/openstack_deploy.LIBERTY is created before executing this play

[Jean-Philippe Evrard (jean-philippe-evrard)]

Could you explain how to reproduce this?

[Jean-Philippe Evrard (jean-philippe-evrard)]

/etc/openstack_deploy.LIBERTY will be created in the deploy-config-changes because copy module is supposed to be recursively copying.

[Bjoern (bjoern-t)]

To reproduce this you can create a empty file at /etc/openstack_deploy.LIBERTY.
As I said, it was a conditional problem and I like to harden the playbooks in a way that they don't break if the destination is there already. Perhaps we just add a check if the destination is a directory

[Andy McCrae (andrew-mccrae)]

Hi BjoernT - just to clarify, a file existed already at "/etc/openstack_deploy.LIBERTY" - this was a file and not a directory, so when the copy task ran it failed to sync because it couldn't create the directory, since a file already existed where the directory should've been.

I'm not too sure what a smart approach would be.

* We could delete the file and replace it with a directory, but we run the risk of removing a potentially important file, we have no real way of telling what the deployer/operator put in that file or why it exists.
* We could fail out if the directory exists as a file - but the end result is the same (the copy fails, nothing copies and the playbook fails out) - so it seems like a pointless check.

I'm not sure what the other solution would be here, if you have some ideas happy to look into that.

[Bjoern (bjoern-t)]

Yes I it was a file, I would propose to add a dynamic timestamp to the backup destination.
If we determine the environment uses a Liberty config, we copy it one time to a /etc/openstack_deploy.LIBERTY.timestamp folder

[Jean-Philippe Evrard (jean-philippe-evrard)]

See discussion here:
http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2016-11-29.log.html#t2016-11-29T17:00:44

[Mohammed Naser (mnaser)]

Bug find against an unsupported old release