After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys

Bug #1536198 reported by Vladimir Khlyunev on 2016-01-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Dmitry Bilunov
7.0.x
High
Dmitry Bilunov
8.0.x
High
Dmitry Bilunov

Bug Description

Discovered on ISO 8.0-429, 7.0 affected at least by steps 8-9.

Steps to reproduce:
1) Deploy any cluster using Fuel
2) Create backup using 'dockerctl backup'
3) Copy the backup file out of Fuel
4) Reinstall Fuel from scratch (e.g. erase mbr and install it from iso)
5) Copy the backup file back to reinstalled Fuel
6) Restore backed up Fuel using 'dockerctl restore'

(issue on step 7 - https://bugs.launchpad.net/fuel/+bug/1536314 )
7) Go to UI and try to verify networks
Result:
Verification failed.
Method verify_networks. Network verification not avaliable because nodes ["1", "2", "3", "4", "5"] not avaliable via mcollective.

8) Bootstrap 1 node
9) Try to connect to the new node via ssh
Result:
[root@nailgun ~]# ssh 10.109.10.4
Warning: Permanently added '10.109.10.4' (ECDSA) to the list of known hosts.
Permission denied (publickey).

(issue will be moved to separate bug after new appearance - locked to reproduce due to failed network check)
9) Deploy changes:
Provision has failed. Failed to execute hook 'shell' command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": ...... (full trace at http://paste.openstack.org/show/484405/ )

10) Generate diagnostic snapshot
Result:
Dump is timed out

I will keep the env, feel free to ask

Fix proposed to branch: master
Review: https://review.openstack.org/270221

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Bilunov (dbilunov)
status: New → In Progress
description: updated
Vladimir Khlyunev (vkhlyunev) wrote :

The bug was not fixed yet - the fix still not megred.

tags: added: team-bugfix
Matthew Mosesohn (raytrac3r) wrote :

This bug should land in time for 8.0 HCF

tags: added: hit-hcf

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/271915
Reason: master first; 270221/5 handles errors and updates only the active bootstrap image

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/271916
Reason: master first; 270221/5 handles errors and updates only the active bootstrap image

Reviewed: https://review.openstack.org/270221
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fce2e2a7649934636338939244723d3c3ea460b5
Submitter: Jenkins
Branch: master

commit fce2e2a7649934636338939244723d3c3ea460b5
Author: Dmitry Bilunov <email address hidden>
Date: Wed Jan 20 17:27:57 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Change-Id: I3945200d02b29ed10e60900f7d03ed30970159a3
    Closes-Bug: #1536198
    DocImpact

Changed in fuel:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/275949

Change abandoned by Dmitry Bilunov (<email address hidden>) on branch: master
Review: https://review.openstack.org/275948
Reason: bad parent

Reviewed: https://review.openstack.org/275949
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c3fc592bbf235985e17eca20b92ae0c1185aa7c8
Submitter: Jenkins
Branch: master

commit c3fc592bbf235985e17eca20b92ae0c1185aa7c8
Author: Dmitry Bilunov <email address hidden>
Date: Thu Feb 4 01:16:50 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Closes-Bug: #1536198
    DocImpact
    Change-Id: I06d8720b05096a2a7955f164d9404ecb28dc9a3a

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/276323
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7574a70511db2ac7d98d820e986cbfec659d6a1e
Submitter: Jenkins
Branch: stable/8.0

commit 7574a70511db2ac7d98d820e986cbfec659d6a1e
Author: Dmitry Bilunov <email address hidden>
Date: Thu Feb 4 01:16:50 2016 +0300

    dockerctl: Update authorized_keys in bootstrap images during restore

    After backup-reinstall-restore Fuel uses bootstrap with wrong ssh keys.

    dockerctl backup procedure does not save the bootstrap's root
    filesystem, which holds the list of authorized keys that should be
    matching the private key located in /root/.ssh on the master.
    It would result in inaccessible nodes from a new master installation
    (which has bootstrap images regenerated with a new key, not matching
    the original one, restored from the backup).

    This patch updates authorized_keys inside squashfsed bootstrap images,
    so it will be retained during backup-reinstall-restore procedure, if
    user has not requested a full backup. Full backups already contain a
    full copy of /var/www/nailgun, so no changes are needed.

    User should carefully restore from non-full backups - only active
    bootstrap will have its ssh keys updated. To propagate ssh keys to
    already running nodes, you can use something like:
      mco rpc --agent execute_shell_command --action execute --arg cmd="echo '$(cat /root/.ssh/id_rsa.pub)' >> /root/.ssh/authorized_keys" -v

    Closes-Bug: #1536198
    DocImpact
    Change-Id: I06d8720b05096a2a7955f164d9404ecb28dc9a3a

tags: added: on-verification

Verification failed accoring to https://bugs.launchpad.net/fuel/+bug/1541539

tags: removed: on-verification
tags: added: on-verification
tags: removed: on-verification
tags: added: on-verification
Artem Panchenko (apanchenko-8) wrote :

Verified on 8.0 RC2/RTM:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

tags: removed: on-verification
tags: added: on-verification
Dmitriy Kruglov (dkruglov) wrote :

Docker removal in 9.0 makes the original scenario not applicable for MOS 9.0.
Verification of the fix will be possible after implementing backup/restore functionality in Octane tooling.

tags: removed: on-verification
Nastya Urlapova (aurlapova) wrote :

Due to Dima's comment tag "dev-to-verify" was added.

tags: added: dev-to-verify
Vladimir Khlyunev (vkhlyunev) wrote :

Actually _this_ issue has no connections with 9.0 backup-restore procedure - so for 9.0 we can "Invalid" this.

tags: added: team-upgrades
removed: dev-to-verify hit-hcf team-bugfix
Maksim Malchuk (mmalchuk) wrote :

marked as 'Fix Released' because we don't have Docker anymore, these fixes were merged before we remove the Docker completely.

Changed in fuel:
status: Fix Committed → Fix Released
no longer affects: fuel/mitaka
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers