overcloud update stack --init-update command can drop state

[Jiří Stránský]

The overcloud update stack --init-update inherits from DeployCommand but can drop/reset some state like roles_data and network_data. We should preserve the state.

[Jiří Stránský (jistr)]

Actually this affects also any files under `user-files` directory of the plan container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/546168

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/546242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543902
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ef23c77ee8a0b03550631f3e40c45d4a786e271e
Submitter: Zuul
Branch: master

commit ef23c77ee8a0b03550631f3e40c45d4a786e271e
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:21:15 2018 +0100

    Add generic methods to upload files to plan

    Either by local filename or by content directly. These will come in
    handy for preserving content on plan update.

    Change-Id: I5dea8c71b86bd35883cbe417e7aba01d69c9ab26
    Partial-Bug: #1749700

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543903
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ba106753ecc3a569bc1c3dbecc34c6f894636ba8
Submitter: Zuul
Branch: master

commit ba106753ecc3a569bc1c3dbecc34c6f894636ba8
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:23:07 2018 +0100

    Add generic method for preserving plan files, use for user-environment.yaml

    This should keep the existing functionality, but via a method which
    could be generically reused for network and roles files too, and maybe
    the plan environment file too.

    Partial-Bug: #1749700
    Change-Id: I4f721b5aa699c6cc23ccb2de8c841caa6b078b30

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546168
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=78e24e21c44fba7e27d11f48f4dc3414b21fd4fd
Submitter: Zuul
Branch: master

commit 78e24e21c44fba7e27d11f48f4dc3414b21fd4fd
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 20 14:37:10 2018 +0100

    Keep original behavior of `overcloud deploy`

    Merging change Ib96f4f5078ec1c45981d051e546bf931d2885ae2 made both
    `overcloud deploy` and `overcloud update stack` try to persist
    user-environment.yaml. But we don't persist `user-files` directory, so
    user-environment.yaml could then point to non-existing files. This
    means both `overcloud deploy` and `overcloud update stack` commands
    are now broken for some cases.

    This patch partially reverts the previous changes. The plan
    persistence logic no longer affects `overcloud deploy`, and only
    focuses on `overcloud update stack`. This way we'll make sure we don't
    introduce backward incompatibility to `overcloud deploy` command at
    this point. Also it might be useful to have a stable `deploy` command
    as a fallback in case the new `overcloud update stack` file
    preservation approach misbehaves.

    And persisting `user-files` directory might have a side effect of
    accumulating garbage files over time (files no longer referenced from
    anywhere), which is another reason to only do that in the `update`
    command for now, not `deploy`. If we implement garbage collection for
    user-files, we could start persisting everything for `deploy`
    too. Alternatively we might also do a complete overhaul of how we
    approach deployment plan on CLI to sync closer with UI.

    Change-Id: I7da71e92af0a6d7ac2d62786d100a1781a4a4fe9
    Partial-Bug: #1749700

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543904
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=d1db364fec1a114a600909cb6afa7490570fec0b
Submitter: Zuul
Branch: master

commit d1db364fec1a114a600909cb6afa7490570fec0b
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:28:52 2018 +0100

    Use the generic preservation method for plan-environment.yaml

    This reuses the previously introduced generic file preservation method
    in a deployment plan to keep plan-environment.yaml too.

    We can also get rid of the separate password keeping, as they are part
    of the plan-environment.yaml.

    Partial-Bug: #1749700
    Change-Id: Ib64296daaa81ff4fb19f1c199ecd6507ab2fb71c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543905
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=8e715b9dea8151025502dffdb77a01127a8707cf
Submitter: Zuul
Branch: master

commit 8e715b9dea8151025502dffdb77a01127a8707cf
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:33:49 2018 +0100

    Preserve roles_data and network_data

    This will make sure that we keep existing roles_data and network_data
    during a plan update, unless the user has passed in new files on the
    command line.

    Partial-Bug: #1749700
    Change-Id: Ic059f50f0b2a3a2b20011851cb0b1576192c0d00

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546242
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=f55ad2c80cc8a193f9c9fc36f3c6085576c708a2
Submitter: Zuul
Branch: master

commit f55ad2c80cc8a193f9c9fc36f3c6085576c708a2
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 20 18:14:07 2018 +0100

    Persist user-files/* for 'overcloud update' command

    Reuse the generic file persistence logic to persist also any files
    under "user-files/" path in the plan. These are files that can be
    potentially referenced from other files in the
    plan (e.g. user-environment.yaml) and if they aren't persisted, the
    plan could get into inconsistent state (broken references).

    Change-Id: I54d10b11e024a481a9ff905a7cce79cb7f562ea0
    Closes-Bug: #1749700

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/549964

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550030

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550032

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550034

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550036

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550030
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=6335326ebb745bbc542d37770edc8d4f60656c90
Submitter: Zuul
Branch: stable/queens

commit 6335326ebb745bbc542d37770edc8d4f60656c90
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:21:15 2018 +0100

    Add generic methods to upload files to plan

    Either by local filename or by content directly. These will come in
    handy for preserving content on plan update.

    Change-Id: I5dea8c71b86bd35883cbe417e7aba01d69c9ab26
    Partial-Bug: #1749700
    (cherry picked from commit ef23c77ee8a0b03550631f3e40c45d4a786e271e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550032
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=f2bc69e2f1d7af48c4a57b99d824abd6258f49d7
Submitter: Zuul
Branch: stable/queens

commit f2bc69e2f1d7af48c4a57b99d824abd6258f49d7
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:23:07 2018 +0100

    Add generic method for preserving plan files, use for user-environment.yaml

    This should keep the existing functionality, but via a method which
    could be generically reused for network and roles files too, and maybe
    the plan environment file too.

    Partial-Bug: #1749700
    Change-Id: I4f721b5aa699c6cc23ccb2de8c841caa6b078b30
    (cherry picked from commit ba106753ecc3a569bc1c3dbecc34c6f894636ba8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550034
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=01111826d2f114f6d733280c78119ee24509fbc9
Submitter: Zuul
Branch: stable/queens

commit 01111826d2f114f6d733280c78119ee24509fbc9
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 20 14:37:10 2018 +0100

    Keep original behavior of `overcloud deploy`

    Merging change Ib96f4f5078ec1c45981d051e546bf931d2885ae2 made both
    `overcloud deploy` and `overcloud update stack` try to persist
    user-environment.yaml. But we don't persist `user-files` directory, so
    user-environment.yaml could then point to non-existing files. This
    means both `overcloud deploy` and `overcloud update stack` commands
    are now broken for some cases.

    This patch partially reverts the previous changes. The plan
    persistence logic no longer affects `overcloud deploy`, and only
    focuses on `overcloud update stack`. This way we'll make sure we don't
    introduce backward incompatibility to `overcloud deploy` command at
    this point. Also it might be useful to have a stable `deploy` command
    as a fallback in case the new `overcloud update stack` file
    preservation approach misbehaves.

    And persisting `user-files` directory might have a side effect of
    accumulating garbage files over time (files no longer referenced from
    anywhere), which is another reason to only do that in the `update`
    command for now, not `deploy`. If we implement garbage collection for
    user-files, we could start persisting everything for `deploy`
    too. Alternatively we might also do a complete overhaul of how we
    approach deployment plan on CLI to sync closer with UI.

    Change-Id: I7da71e92af0a6d7ac2d62786d100a1781a4a4fe9
    Partial-Bug: #1749700
    (cherry picked from commit 78e24e21c44fba7e27d11f48f4dc3414b21fd4fd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550036
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=90018649463439e6adc37f1508c9b5eeecd12199
Submitter: Zuul
Branch: stable/queens

commit 90018649463439e6adc37f1508c9b5eeecd12199
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:28:52 2018 +0100

    Use the generic preservation method for plan-environment.yaml

    This reuses the previously introduced generic file preservation method
    in a deployment plan to keep plan-environment.yaml too.

    We can also get rid of the separate password keeping, as they are part
    of the plan-environment.yaml.

    Partial-Bug: #1749700
    Change-Id: Ib64296daaa81ff4fb19f1c199ecd6507ab2fb71c
    (cherry picked from commit d1db364fec1a114a600909cb6afa7490570fec0b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550038
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e8e1484fe77198fa2d3fb5ed1ca5f99961368b5d
Submitter: Zuul
Branch: stable/queens

commit e8e1484fe77198fa2d3fb5ed1ca5f99961368b5d
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 13 12:33:49 2018 +0100

    Preserve roles_data and network_data

    This will make sure that we keep existing roles_data and network_data
    during a plan update, unless the user has passed in new files on the
    command line.

    Partial-Bug: #1749700
    Change-Id: Ic059f50f0b2a3a2b20011851cb0b1576192c0d00
    (cherry picked from commit 8e715b9dea8151025502dffdb77a01127a8707cf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549964
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=0f505a0510909dd2e696501ea7b8bd04fc5408c6
Submitter: Zuul
Branch: stable/queens

commit 0f505a0510909dd2e696501ea7b8bd04fc5408c6
Author: Jiri Stransky <email address hidden>
Date: Tue Feb 20 18:14:07 2018 +0100

    Persist user-files/* for 'overcloud update' command

    Reuse the generic file persistence logic to persist also any files
    under "user-files/" path in the plan. These are files that can be
    potentially referenced from other files in the
    plan (e.g. user-environment.yaml) and if they aren't persisted, the
    plan could get into inconsistent state (broken references).

    Change-Id: I54d10b11e024a481a9ff905a7cce79cb7f562ea0
    Closes-Bug: #1749700
    (cherry picked from commit f55ad2c80cc8a193f9c9fc36f3c6085576c708a2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 10.0.0 release.