Resetting and redeploying Openstack environment fails with copy_haproxy_keys task error

[Andres Toomsalu]

It seems that on environment reset /var/lib/fuel/keys/{CLUSTER_ID}/haproxy/public_haproxy.* files are removed - but on environment redeploy generate_haproxy_keys task is skipped and hence copy_haproxy_keys task fails (as no source files exist).

Fuel version:
yum list installed | grep fuel
fuel.noarch 9.0.0-1.mos6357 @mos9.0-updates
fuel-agent.noarch 9.0.0-1.mos291 @mos9.0-updates
fuel-bootstrap-cli.noarch 9.0.0-1.mos291 @mos9.0-updates
fuel-library9.0.noarch 9.0.0-1.mos8606 @mos9.0-updates
fuel-migrate.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuel-mirror.noarch 9.0.0-1.mos154 @mos9.0-updates
fuel-misc.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuel-nailgun.noarch 9.0.0-1.mos8861 @mos9.0-updates
fuel-notify.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuel-openstack-metadata.noarch 9.0.0-1.mos8861 @mos9.0-updates
fuel-ostf.noarch 9.0.0-1.mos946 @mos9.0-updates
fuel-provisioning-scripts.noarch 9.0.0-1.mos8861 @mos9.0-updates
fuel-release.noarch 9.0.0-1.mos6357 @mos9.0-updates
fuel-setup.noarch 9.0.0-1.mos6357 @mos9.0-updates
fuel-ui.noarch 9.0.0-1.mos2814 @mos9.0-updates
fuel-utils.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuelmenu.noarch 9.0.0-1.mos275 @mos9.0-updates
python-fuelclient.noarch 9.0.0-1.mos356 @mos9.0-updates

Workaround is to manually run:
sh /etc/puppet/modules/osnailyfacter/modular/astute/generate_haproxy_keys.sh -i <env-id> -h <dashboard/api fqdn> -o 'haproxy' -p /var/lib/fuel/keys/

...and then redeploy environment.

[Oleksii Aleksieiev (alexzzman)]

I have encountered the same problem. Fuel 9.1.

Looks like the generate_haproxy_keys task and some other tasks are skipped after reset.

The workaround for this is to delete all tasks related to this environment after reset.

Step to reproduce:
Create and configure environment with self-signed ssl enabled.
Start deployment and make it failed during deployment or stop deployment.
Reset environment.
Try to deploy again.
Expected behavior is to get successful deployment.
Received result: task copy_haproxy_keys is failed.

[Stanislaw Bogatkin (sbogatkin)]

Hi guys,
please, attach your diagnostic snapshot with logs. I have been tested this scenario on 9.0 and it works flawlessly - keys are deleted after cluster reset and generate_keys task also run after cluster redeploy. As as result, redeploy also works okay.

[Stanislaw Bogatkin (sbogatkin)]

I've got this reproduced. Problem is that we store deployment info data after reset. It leads to return non-null value for old($) in yaql expressions and as a result all the tasks which has been ran based on this value are broken.

So we need to clear all previoud cluster info after cluster reset.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/414089

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/414096

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/414097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414096
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=64e25bf850f61c8453407077c67df6c0abc2f9bf
Submitter: Jenkins
Branch: master

commit 64e25bf850f61c8453407077c67df6c0abc2f9bf
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 22 16:03:56 2016 +0300

    Use TransactionCollection when deleting tasks after cluster reset

    TaskCollection doesn't contains all the tasks needed to be deleted on
    cluster reset, so use TransactionCollection instead.

    Change-Id: I49077a3f6402fcfdb1f71e85fd48602734f1923f
    Closes-Bug: #1649987
    Co-Authored-With: Bulat Gaifullin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414097
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=095fa2a74bd8f034a03abb18f4990c4a33833301
Submitter: Jenkins
Branch: stable/newton

commit 095fa2a74bd8f034a03abb18f4990c4a33833301
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 22 16:03:56 2016 +0300

    Use TransactionCollection when deleting tasks after cluster reset

    TaskCollection doesn't contains all the tasks needed to be deleted on
    cluster reset, so use TransactionCollection instead.

    Change-Id: I49077a3f6402fcfdb1f71e85fd48602734f1923f
    Closes-Bug: #1649987
    Co-Authored-With: Bulat Gaifullin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414089
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=b61cba6d74ed42326992333cccf548da1bb51379
Submitter: Jenkins
Branch: stable/mitaka

commit b61cba6d74ed42326992333cccf548da1bb51379
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 22 15:49:23 2016 +0300

    Use TransactionCollection when deleting tasks after cluster reset

    TaskCollection doesn't contains all the tasks needed to be deleted on
    cluster reset, so use TransactionCollection instead.

    Change-Id: I49077a3f6402fcfdb1f71e85fd48602734f1923f
    Closes-Bug: #1649987
    Co-Authored-With: Bulat Gaifullin <email address hidden>

[Ekaterina Shutova (eshutova)]

Verified on 9.2 snapshot #742.
Deployment is successfully completed after env reset w/o any errors.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-web 11.0.0.0rc1 release candidate.

[Vladimir Khlyunev (vkhlyunev)]

1575 - reset & redeploy works fine