After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys

[Vladimir Khlyunev]

Discovered on ISO 8.0-429, 7.0 affected at least by steps 8-9.

Steps to reproduce:
1) Deploy any cluster using Fuel
2) Create backup using 'dockerctl backup'
3) Copy the backup file out of Fuel
4) Reinstall Fuel from scratch (e.g. erase mbr and install it from iso)
5) Copy the backup file back to reinstalled Fuel
6) Restore backed up Fuel using 'dockerctl restore'

(issue on step 7 - https://bugs.launchpad.net/fuel/+bug/1536314 )
7) Go to UI and try to verify networks
Result:
Verification failed.
Method verify_networks. Network verification not avaliable because nodes ["1", "2", "3", "4", "5"] not avaliable via mcollective.

8) Bootstrap 1 node
9) Try to connect to the new node via ssh
Result:
[root@nailgun ~]# ssh 10.109.10.4
Warning: Permanently added '10.109.10.4' (ECDSA) to the list of known hosts.
Permission denied (publickey).

(issue will be moved to separate bug after new appearance - locked to reproduce due to failed network check)
9) Deploy changes:
Provision has failed. Failed to execute hook 'shell' command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": ...... (full trace at http://paste.openstack.org/show/484405/ )

10) Generate diagnostic snapshot
Result:
Dump is timed out

I will keep the env, feel free to ask

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/270221

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/271915

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/271916

[Vladimir Khlyunev (vkhlyunev)]

The bug was not fixed yet - the fix still not megred.

[Matthew Mosesohn (raytrac3r)]

This bug should land in time for 8.0 HCF

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/271915
Reason: master first; 270221/5 handles errors and updates only the active bootstrap image

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/271916
Reason: master first; 270221/5 handles errors and updates only the active bootstrap image

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270221
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fce2e2a7649934636338939244723d3c3ea460b5
Submitter: Jenkins
Branch: master

commit fce2e2a7649934636338939244723d3c3ea460b5
Author: Dmitry Bilunov <email address hidden>
Date: Wed Jan 20 17:27:57 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Change-Id: I3945200d02b29ed10e60900f7d03ed30970159a3
    Closes-Bug: #1536198
    DocImpact

[Bogdan Dobrelya (bogdando)]

Reverting by https://review.openstack.org/#/c/275825/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275948

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275949

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: master
Review: https://review.openstack.org/275948
Reason: bad parent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/276323

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275949
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c3fc592bbf235985e17eca20b92ae0c1185aa7c8
Submitter: Jenkins
Branch: master

commit c3fc592bbf235985e17eca20b92ae0c1185aa7c8
Author: Dmitry Bilunov <email address hidden>
Date: Thu Feb 4 01:16:50 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Closes-Bug: #1536198
    DocImpact
    Change-Id: I06d8720b05096a2a7955f164d9404ecb28dc9a3a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276323
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7574a70511db2ac7d98d820e986cbfec659d6a1e
Submitter: Jenkins
Branch: stable/8.0

commit 7574a70511db2ac7d98d820e986cbfec659d6a1e
Author: Dmitry Bilunov <email address hidden>
Date: Thu Feb 4 01:16:50 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Closes-Bug: #1536198
    DocImpact
    Change-Id: I06d8720b05096a2a7955f164d9404ecb28dc9a3a

[Artem Hrechanychenko (agrechanichenko)]

Verification failed accoring to https://bugs.launchpad.net/fuel/+bug/1541539

[Artem Panchenko (apanchenko-8)]

Verified on 8.0 RC2/RTM:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Dmitriy Kruglov (dkruglov)]

Docker removal in 9.0 makes the original scenario not applicable for MOS 9.0.
Verification of the fix will be possible after implementing backup/restore functionality in Octane tooling.

[Nastya Urlapova (aurlapova)]

Due to Dima's comment tag "dev-to-verify" was added.

[Vladimir Khlyunev (vkhlyunev)]

Actually _this_ issue has no connections with 9.0 backup-restore procedure - so for 9.0 we can "Invalid" this.

[Maksim Malchuk (mmalchuk)]

marked as 'Fix Released' because we don't have Docker anymore, these fixes were merged before we remove the Docker completely.