9.0 to 9.2 upgrade fails when Fuel admin password was changed

Bug #1681192 reported by Robert Jesacher
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
Sergii Rizvan

Bug Description

Detailed bug description:
if the default password in the Fuel WebUI is changed, the upgrade process to 9.2 fails.

Steps to reproduce:
* fresh install of the Fuel-master (version 9.0)
* only changed the default fuel admin password.
* also changed it in "/root/.config/fuel/fuel_client.yaml" to be able to do a "fuel2 fuel-version"
* then tried to proceed with the update as mentioned on the mirantis-page.
* the step "ansible-playbook playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}'" fails

Expected results:
-----------------------------
[root@fuel mos_mu]# ansible-playbook playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}'

PLAY [fuel] ********************************************************************

TASK [Make backup directory exists on Fuel] ************************************
[-------- snip --------]

TASK [Upgrade kernel data on Fuel] *********************************************
changed: [fuel.foobar]

PLAY RECAP *********************************************************************
fuel.foobar : ok=1 changed=1 unreachable=0 failed=0
-----------------------------

Actual result:
-----------------------------
[root@fuel mos_mu]# ansible-playbook playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}'

PLAY [fuel] ********************************************************************

TASK [Make backup directory exists on Fuel] ************************************
ok: [fuel.foobar]

TASK [Stop docker containers] **************************************************
skipping: [fuel.foobar]

TASK [Make a backup of docker containers data] *********************************
skipping: [fuel.foobar]

TASK [Yum update] **************************************************************
skipping: [fuel.foobar]

TASK [Load new Fuel images] ****************************************************
skipping: [fuel.foobar]

TASK [Destroy docker containers] ***********************************************
skipping: [fuel.foobar]

TASK [Start docker containers] *************************************************
skipping: [fuel.foobar]

TASK [Make a backup of docker containers data] *********************************
changed: [fuel.foobar]
 [WARNING]: Consider using unarchive module rather than running tar

TASK [Yum clean all] ***********************************************************
changed: [fuel.foobar]
 [WARNING]: Consider using yum module rather than running yum

TASK [Yum update] **************************************************************
changed: [fuel.foobar]

TASK [Systemd loads fresh units] ***********************************************
changed: [fuel.foobar]

TASK [Re-apply puppet master node configuration] *******************************
fatal: [fuel.foobar]: FAILED! => {"changed": true, "cmd": ["/etc/puppet/modules/fuel/examples/deploy.sh"], "delta": "0:05:19.920458", "end": "2017-04-08 21:07:15.012527", "failed": true, "rc": 1, "start": "2017-04-08 21:01:55.092069", "stderr": "Warning: Scope(Class[Rabbitmq]): $ssl_stomp_port requires that $ssl => true and will be ignored\nWarning: Scope(Class[Keystone]): Keystone under Eventlet has been deprecated during the Kilo cycle. Support for deploying under eventlet will be dropped as of the M-release of OpenStack.\nWarning: Puppet::Type::Keystone_tenant::ProviderOpenstack: Support for a resource without the domain set is deprecated in Liberty cycle. It will be dropped in the M-cycle. Currently using 'Default' as default domain name while the default domain id is '6319e917c94146a592d40b6cfec91e86'.\nError: fuel rel --sync-deployment-tasks --dir /etc/puppet/ returned 1 instead of one of [0]\nError: /Stage[main]/Main/Exec[sync_deployment_tasks]/returns: change from notrun to 0 failed: fuel rel --sync-deployment-tasks --dir /etc/puppet/ returned 1 instead of one of [0]", "stdout": "##################################\nDEPLOYMENT TASK: hiera\n##################################\nDebug: Runtime environment: puppet_version=3.8.5, ruby_version=2.0.0, run_mode=user, default_encoding=UTF-8\nDebug: Evicting cache entry for environment 'production'\nDebug: Caching environment 'production' (ttl = 0 sec)\nDebug: Evicting cache entry for environment 'production'\nDebug: Caching environment 'production' (ttl = 0 sec)\nDebug: Loading external facts from /etc/puppet/modules/openstacklib/facts.d\nDebug: Loading external facts from /var/lib/puppet/facts.d\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/mysql/lib/facter/mysql_version.rb\nDebug: Loading facts from /etc/puppet/modules/mysql/lib/facter/mysql_server_id.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/openstacklib/lib/facter/os_service_default.rb\nDebug: Loading facts from /etc/puppet/modules/openstacklib/lib/facter/os_package_type.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/staging/lib/facter/staging_windir.rb\nDebug: Loading facts from /etc/puppet/modules/staging/lib/facter/staging_http_get.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb\nDebug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb\nDebug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/pe_version.rb\nDebug: Loading facts from /etc/puppet/modules/stdlib/lib/facter/root_home.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/l23_os.rb\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/check_kern_module.rb\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/default_route.rb\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/ovs_vlan_splinters.rb\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/openvswitch.rb\nDebug: Loading facts from /etc/puppet/modules/l23network/lib/facter/fqdn_hostname.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/vswitch/lib/facter/pci_address.rb\nDebug: Loading facts from /etc/puppet/modules/vswitch/lib/facter/ovs.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/osnailyfacter/lib/facter/mounts.rb\nDebug: Loading facts from /etc/puppet/modules/osnailyfacter/lib/facter/naily.rb\nDebug: Loading facts from /etc/puppet/modules/osnailyfacter/lib/facter/libvirt_package_version.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/pacemaker/lib/facter/pcmk_node_name.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/apt/lib/facter/apt_update_last_success.rb\nDebug: Loading facts from /etc/puppet/modules/apt/lib/facter/apt_updates.rb\nDebug: Loading facts from /etc/puppet/modules/apt/lib/facter/apt_reboot_required.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/iops.rb\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/fuel_openstack_version.rb\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/fuel_release.rb\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/acpid_version.rb\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/acpi_event.rb\nDebug: Loading facts from /etc/puppet/modules/fuel/lib/facter/generate_fuel_key.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/openstack_tasks/lib/facter/allocated_hugepages.rb\nInfo: Loading facts\nDebug: Loading facts from /etc/puppet/modules/firewall/lib/facter/ip6tables_version.rb\nDebug: Loading facts from /etc/puppet/modules/firewall/lib/facter/iptables_version.rb\n
[---- snip (goes on for several hundred lines like that) ----]
-----------------------------

Reproducibility:
tried with existing install & fresh install of fuel master-host. both times i saw the error.

Workaround:
change the admin password in the WebUI to the default value.

Impact:
upgrade to 9.2 is not possible.

Additional information:
I saw that the password in /root/.config/fuel/fuel_client.yaml was changed back to the default one after running the above mentioned playbook.
after changing the password in the WebUI to the default, and executing "ansible-playbook playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}'" again, it ran through without an issue.

If this is a general issue, you could mention that in the documentation (as a quick "fix")
or
1) the error message I posted is really misleading, as it doesn't simply state "password wrong" -it took me quite some time to find that.
2) find a way to deal with the changed password, that doesn't break the upgrade-process

thanks for the otherwise excellent work!
best regards,
Robert

Tags: area-docs
Changed in mos:
milestone: none → 9.2-mu-2
assignee: nobody → MOS Maintenance (mos-maintenance)
importance: Undecided → Medium
status: New → Confirmed
tags: added: area-docs
Sergii Rizvan (srizvan)
Changed in mos:
assignee: MOS Maintenance (mos-maintenance) → Sergii Rizvan (srizvan)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to mos/mos-docs (stable/9.2)

Fix proposed to branch: stable/9.2
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/35627

Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to mos/mos-docs (stable/9.2)

Reviewed: https://review.fuel-infra.org/35627
Submitter: Olena Logvinova <email address hidden>
Branch: stable/9.2

Commit: f9b646666963b4ac3b838504dbd613d296f4f8ea
Author: Sergii Rizvan <email address hidden>
Date: Wed Jun 14 15:44:50 2017

[RN 9.2] Added a step to the prerequisites section of the update procedure

Before updating Fuel Master node, user should that check correct
admin password is present in the '/etc/fuel/astute.yaml' file.
This step is added to the prerequisites section of the update procedure.

Change-Id: I338ef86554181db1415ac827b0737c956a92520a
Closes-Bug: #1681192

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on 9.2 MU 2 (MOS_UBUNTU_ID = 9.0-2017-06-13-142421)

Changed in mos:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.