[9.0] mos9.0-updates mirror may cause cluster LCM problems

[Victor Ryzhenkin]

Detailed bug description:
 Since 9.1 was released and packages was uploaded to mos9.0-updates mirror repo, now 9.0 ISO installation comes with updated packages, even mos9.0-updates repo disabled.
 This happens because master-node turns on mos9.0-* repos before master-node puppet run.
 Because of this, some packages on Fuel master comes from mos9.0-updates instead of @mos local mirror. This can spawn problems like 'dpdk' error. Reproducing below.
Steps to reproduce:
 1. Deploy any 9.0 cluster.
 2. Make changes.
 3. Deploy changes.
Expected results:
 Cluster deployed.
Actual result:
 Deployment error occurred. Reason: AssertionError: Task 'deploy' has incorrect status. error != ready, 'u'dpdk''
Reproducibility:
 Always
Workaround:
 Possible workarounds(not verified):
 1. Reinstall fuel-nailgun.noarch
 2. Patch code as done in this patch (https://bugs.launchpad.net/fuel/+bug/1616119)
Impact:
 By default, there are no opportunity to change deployed cluster.
Additional information:
 [root@nailgun ~]# yum list installed | grep fuel
fuel.noarch 9.0.0-1.mos6349 @mos
fuel-agent.noarch 9.0.0-1.mos285 @mos
fuel-bootstrap-cli.noarch 9.0.0-1.mos285 @mos
fuel-library9.0.noarch 9.0.0-1.mos8460 @mos
fuel-migrate.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuel-mirror.noarch 9.0.0-1.mos140 @mos
fuel-misc.noarch 9.0.0-1.mos8460 @mos
fuel-nailgun.noarch 9.0.0-1.mos8861 @mos9.0-updates
fuel-notify.noarch 9.0.0-1.mos8607 @mos9.0-updates
fuel-openstack-metadata.noarch 9.0.0-1.mos8743 @mos
fuel-ostf.noarch 9.0.0-1.mos946 @mos9.0-updates
fuel-provisioning-scripts.noarch 9.0.0-1.mos8743 @mos
fuel-release.noarch 9.0.0-1.mos6349 @mos
fuel-setup.noarch 9.0.0-1.mos6349 @mos
fuel-ui.noarch 9.0.0-1.mos2717 @mos
fuel-utils.noarch 9.0.0-1.mos8460 @mos
fuelmenu.noarch 9.0.0-1.mos274 @mos
python-fuelclient.noarch 9.0.0-1.mos325 @mos

[Oleksiy Molchanov (omolchanov)]

Doc team, can you check this, if it is correct, please reassign to fuel-qa.

[Nastya Urlapova (aurlapova)]

@Oleksey, why did you reassign it to QA team, we don't support mechanism of updates!

[Vladimir Khlyunev (vkhlyunev)]

docs team we need confirmation that our fuel installation guide includes "update from 9.0 to 9.1" step as _mandatory_ requirement. For now I see that at [0] it mentioned as "see also", not "do it any installation"

[0] http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/install_install_fuel.html

[Olena Logvinova (ologvinova)]

@vkhlyunev, the link you've provided is for the upstream Fuel version where update to the latest Mitaka is _not mandatory_ (unless I miss smth).

As for MOS, the Quickstart guide [0] contains the update step (#4) as a mandatory one.

BUT, since the update procedure for _existing_envs_ [1] contains a warning (step 5) about customizations' loss, I am not sure whether we can make this procedure _mandatory_ (unless we rephrase the warning).

@vsedelnik, @dmeltsaykin, could you please comment on this?

[0] https://docs.mirantis.com/openstack/fuel/fuel-9.1/quickstart-guide/qs-install-scripts-procedure.html

[1] https://docs.mirantis.com/openstack/fuel/fuel-9.1/release-notes/update-product/update-env-9-9-1.html

[Denis Meltsaykin (dmeltsaykin)]

1. This bug has nothing to do with packages from 9.1 being fetched from updates while 9.0 from ISO is being installed, since the bug is in the fuel-library's 9.0 package.

2. The update from 9.0 to 9.1 is not mandatory, IMO. How can it be mandatory, if it is up to a user making such decisions? From my point of view, there is a bug in fuel-library of 9.0, which prevents from deploying changes. The fix of this bug is included into 9.1, if you want this fixed you literally have two options: either update to 9.1 or manually patch fuel-library. That is, nothing is mandatory.

[Olena Logvinova (ologvinova)]

Thanks Denis!

We can describe this bug in Known issues section of RN 9.0 [0]. But until the **Workaround** is verified and confirmed, marking the bug as Incomplete.

[0] https://docs.mirantis.com/openstack/fuel/fuel-9.0/release-notes.html#known-issues

[Vladimir Khlyunev (vkhlyunev)]

@ologvinova
there is 3 workarounds:
1 - update to 9.1 right after fuel installation
2 - disable mos9.0-updates repository at "fuelmenu" step by direct ssh to node and "yum-config-manager --disable mos9.0-updates --save"
3 - downgrade 4 packages and re-run migration script

the first and the second was already verified, we only need to find proper set of commands for the third one. Will try to do this (or @vryzhenkin can help with it)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/398853

[Nastya Urlapova (aurlapova)]

@Olena, what is a reason for Incomplete stateof issue, you cannot prepare review with ways of update?

[Olena Logvinova (ologvinova)]

@Nastya, before Vladimir provided workarounds today, the description was incomplete, bc until today, the workaround was not verified (therefore, the bug was Incomplete).

And still, while the 3rd workaround verification is still in progress, I see no reason in preparing an incomplete procedure.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/9.1
Change author: Olena Logvinova <email address hidden>
Review: https://review.fuel-infra.org/28608

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/9.0
Change author: Olena Logvinova <email address hidden>
Review: https://review.fuel-infra.org/28610

[Olena Logvinova (ologvinova)]

As discussed offline with Vladimir, the 3rd workaround won't be published since it is too risky and time-consuming for the testing.

Adding the bug description to both versions of RN - 9.0 and 9.1.

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Olena Logvinova <email address hidden> on branch: stable/9.1
Review: https://review.fuel-infra.org/28608

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/28610
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.0

Commit: bcd34a99c06b40254ea770d2b0706fa2800ddcd6
Author: Olena Logvinova <email address hidden>
Date: Mon Nov 21 10:30:31 2016

[RN 9.0] Add LP1641931 to the Fuel known issues list

Change-Id: I3526b129da70c87a4a80ef003b93eac293962665
Closes-Bug: #1641931

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #541.

LP1641931 was added into 'Known issues' section of 'Release Notes' in https://infra-ci.fuel-infra.org/job/verify-mos-docs/1682/MOS_Docs_build_results/. Also bug LP1641931 appeared in https://docs.mirantis.com/openstack/fuel/fuel-9.0/release-notes.html#known-issues.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/398853
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=8ef1ad1fac216774d78a3a9c9a9281624a731ba4
Submitter: Jenkins
Branch: stable/mitaka

commit 8ef1ad1fac216774d78a3a9c9a9281624a731ba4
Author: Vladimir Khlyunev <email address hidden>
Date: Thu Nov 17 10:59:31 2016 +0300

    Allow fuel-qa to install clean 9.0 Fuel master node

    Nailgun package is installing by puppet manifests which are running
    when mos9.0-updates repo is enabled by bootstrap_admin_node.sh
    We have to disable it again for using local fuel's repo.
    Also move restriction for FORCE_DISABLE_UPDATES with UPDATE_MASTER
    to init method.

    Change-Id: I9828f4741457b269e82e1eb128f6332dab8f6b61
    Related-bug:1641931

[Maria Zlatkova (mzlatkova)]

This bug has been described in release notes 9.0.