db-migration: if instance.attributes.generated['deployed_before']['value'] != value: KeyError: 'deployed_before'

[Ekaterina Khomyakova]

Steps to reproduce:
1. Deploy environment with any settings using 7.0 Fuel
2. Upgrade fuel to 8.0 (ISO #570) using fuel-octane
3. Add Controller to the existing environment
4. Deploy the environment
5. Wait before all nodes will ready

Observed behavior:
All nodes are ready but task stays in progress.

Root cause:
After deployment astute reports to "deploy_resp" RPC method but:

 File "/usr/lib/python2.7/site-packages/nailgun/objects/cluster.py", line 366, in set_deployed_before_flag
 if instance.attributes.generated['deployed_before']['value'] != value: KeyError: 'deployed_before'
(full trace http://paste.openstack.org/show/489100/ )

Looks like db migration is broken and it does not updated required field

[Vladimir Khlyunev (vkhlyunev)]

RCA: /etc/cobbler/authorized_keys in cobbler's docker was not updated with restored SSH public key.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286438

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/286438
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=f78d006175cf9c158d3c196702b1db1a4d234ddf
Submitter: Jenkins
Branch: master

commit f78d006175cf9c158d3c196702b1db1a4d234ddf
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/287303

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/287304

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287303
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3105a92a3bda61d7addbccf943e4f0daddb43165
Submitter: Jenkins
Branch: stable/8.0

commit 3105a92a3bda61d7addbccf943e4f0daddb43165
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339
    (cherry picked from commit f78d006175cf9c158d3c196702b1db1a4d234ddf)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/287304
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=2cf3c5dfe72c0db70a9be82b459df6185fe8b93d
Submitter: Jenkins
Branch: stable/7.0

commit 2cf3c5dfe72c0db70a9be82b459df6185fe8b93d
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339
    (cherry picked from commit f78d006175cf9c158d3c196702b1db1a4d234ddf)

[Vladimir Khlyunev (vkhlyunev)]

Issue still exists

[Oleg S. Gelbukh (gelbuhos)]

Is it correct to assume that, since it is the DB upgrade issue, it does not affect version 7.0?

[Vladimir Khlyunev (vkhlyunev)]

Yes, 7.0 does not affected

[Oleg S. Gelbukh (gelbuhos)]

Note that this issue effectively renders the env unmanageable after the upgrade+scale-out operation.

[Oleg S. Gelbukh (gelbuhos)]

We implemented a workaround for this bug in upgrade automation scripts for 8.0 and 9.0. However, this issue still needs to be fixed in Nailgun code base and should be back ported to version 8.0-mu1.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/292804

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292804
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=cfa146e9dd7d1579fc4baa7580aedb8afb57414c
Submitter: Jenkins
Branch: master

commit cfa146e9dd7d1579fc4baa7580aedb8afb57414c
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1551339

[Rodion Tikunov (rtikunov)]

Please, provide here the link on documentation with instruction for how to upgrade MOS from 7.0 to 8.0. It needs for reproducing this bug.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/8.0
Change author: Artem Roma <email address hidden>
Review: https://review.fuel-infra.org/21257

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/321414

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Rodion Tikunov <email address hidden> on branch: stable/8.0
Review: https://review.fuel-infra.org/21257
Reason: Wrong repo

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321414
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=48ee65f15e346cff46e0bf0cfba9dbe9afe43528
Submitter: Jenkins
Branch: stable/8.0

commit 48ee65f15e346cff46e0bf0cfba9dbe9afe43528
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1551339
    (cherry picked from commit cfa146e9dd7d1579fc4baa7580aedb8afb57414c)

[Ekaterina Shutova (eshutova)]

Verified on MOS 8.0 + MU3 updates.
Problem was not reproduced since some workaround was already merged to fuel-octane.
Steps from description were used for checking fix.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/340827