Fuel for OpenStack

Cannot retrieve repository metadata (repomd.xml) for repository: auxiliary

Bug #1480282 reported by Maksym Strukov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Ihor Kalnytskyi
Fuel for OpenStack 7.0

Bug Description

Steps:
            1. Revert deploy_neutron_gre snapshot
            2. Upgrade with rollback
            3. Run OSTF
            4. Run network verification
            5. Upgrade fuel master

Actual result:
Upgrade failed:

2015-07-31 05:37:40 DEBUG 6464 (utils) Copying "/var/upgrade/repos/2015.1.0-7.0" -> "/var/www/nailgun/2015.1.0-7.0" [overwrite=1 symlinks=1]
2015-07-31 05:37:48 DEBUG 6464 (utils) Render template from /var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/templates/nailgun.repo to /etc/yum.repos.d/7.0_nailgun.repo with params: {'gpgcheck': 0, 'skip_if_unavailable': 0, 'name': '7.0_nailgun', 'baseurl': 'file:/var/www/nailgun/2015.1.0-7.0/centos/x86_64'}
2015-07-31 05:37:48 DEBUG 6464 (utils) Execute command "yum clean all"
2015-07-31 05:37:48 DEBUG 6464 (utils) Stdout and stderr of command "yum clean all":
2015-07-31 05:37:48 DEBUG 6464 (utils) Loaded plugins: fastestmirror, priorities
2015-07-31 05:37:48 DEBUG 6464 (utils) Cleaning repos: 7.0_nailgun auxiliary mos6.1-security mos6.1-updates nailgun
2015-07-31 05:37:48 DEBUG 6464 (utils) Cleaning up Everything
2015-07-31 05:37:48 DEBUG 6464 (utils) Cleaning up list of fastest mirrors
2015-07-31 05:37:48 DEBUG 6464 (utils) Command "yum clean all" successfully executed
2015-07-31 05:37:48 DEBUG 6464 (utils) Execute command "yum install -v -y fuel-7.0.0"
2015-07-31 05:37:48 DEBUG 6464 (utils) Stdout and stderr of command "yum install -v -y fuel-7.0.0":
2015-07-31 05:37:49 DEBUG 6464 (utils) Loading "fastestmirror" plugin
2015-07-31 05:37:49 DEBUG 6464 (utils) Loading "priorities" plugin
2015-07-31 05:37:49 DEBUG 6464 (utils) Not loading "product-id" plugin, as it is disabled
2015-07-31 05:37:49 DEBUG 6464 (utils) Not loading "subscription-manager" plugin, as it is disabled
2015-07-31 05:37:49 DEBUG 6464 (utils) Config time: 0.006
2015-07-31 05:37:49 DEBUG 6464 (utils) Yum Version: 3.2.29
2015-07-31 05:37:49 DEBUG 6464 (utils) Setting up Package Sacks
2015-07-31 05:37:49 DEBUG 6464 (utils) Determining fastest mirrors
2015-07-31 05:37:49 DEBUG 6464 (utils) file:///var/www/nailgun/2015.1.0-7.0/centos/auxiliary/repodata/repomd.xml: [Errno 14] Could not open/read file:///var/www/nailgun/2015.1.0-7.0/centos/auxiliary/repodata/repomd.xml
2015-07-31 05:37:49 DEBUG 6464 (utils) Trying other mirror.
2015-07-31 05:37:49 DEBUG 6464 (utils) Error: Cannot retrieve repository metadata (repomd.xml) for repository: auxiliary. Please verify its path and try again
2015-07-31 05:37:49 ERROR 6464 (upgrade) HostSystemUpgrader: failed to upgrade: "Shell command executed with "1" exit code: yum install -v -y fuel-7.0.0 "
Traceback (most recent call last):
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 97, in upgrade
    self.install_packages()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 143, in install_packages
    utils.exec_cmd('yum install -v -y {0}'.format(package))
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: yum install -v -y fuel-7.0.0
2015-07-31 05:37:49 ERROR 6464 (upgrade) *** UPGRADE FAILED

Env:
6.1-525 -> 7.0-113

Revision history for this message
Maksym Strukov (unbelll) wrote :
Andrey Sledzinskiy (asledzinskiy)
Changed in fuel:
assignee: nobody → Fuel Python Team (fuel-python)
milestone: none → 7.0
Artem Roma (aroma-x)
Changed in fuel:
status: New → Confirmed
Igor Marnat (imarnat)
tags: added: fuel-to-mos
Alexander Tsamutali (astsmtl)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Alexander Tsamutali (astsmtl)
Ihor Kalnytskyi (ikalnytskyi)
Changed in fuel:
assignee: Alexander Tsamutali (astsmtl) → Igor Kalnitsky (ikalnitsky)
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Well, the problem is the following:

* In 6.1 we have the /etc/yum.repos.d/auxiliary.repo points to file:///var/www/nailgun/2014.2.2-6.1/centos/auxiliary/
* When we perform a master node upgrade from 6.1 to 7.0, the puppet manifest replaces the /etc/yum.repos.d/auxiliary.repo to points it to file:///var/www/nailgun/2015.1.0-7.0/centos/auxiliary/
* In case of upgrade fail the rollback procedure is initiated.
* When we start upgrade procedure once again, it fails because yum is unable to fetch file:///var/www/nailgun/2015.1.0-7.0/centos/auxiliary/ repo (it was removed during rollback procedure).

I don't know how to fix it's better. I see two way to fix it:

1/ Move auxiliry repo management to fuel_upgrade. That means, we shouldn't create auxiliary.repo file in puppet manifests, and should create/remove this file in fuel_upgrade script.

2/ Create a 7.0_auxiliary.repo in puppet manifests, and add hack to remove this file in fuel_upgrade script (similar to this one https://github.com/stackforge/fuel-web/blob/master/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/engines/host_system.py#L158-L163 )

I think anyway, in case of upgrading we should create not "auxiliary.repo", but "7.0_auxiliar.repo", just like we do for the "nailgun.repo".

Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Ok, as we discussed with some fuelers it's better to do the following things for 7.0:

1/ Fix puppet manifest to create 7.0_auxiliary during upgrade master node case
2/ Remove 7.0_* repo file during rollback in fuel_upgrade script.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211252

Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Matthew Mosesohn (raytrac3r)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211589

Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Igor Kalnitsky (ikalnitsky)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/211252
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=eab7b68399cf2249c4f081ae850a21c1b97fcf7c
Submitter: Jenkins
Branch: master

commit eab7b68399cf2249c4f081ae850a21c1b97fcf7c
Author: Matthew Mosesohn <email address hidden>
Date: Mon Aug 10 19:29:02 2015 +0300

    Create auxiliary repo with fuel version

    Fuel 7.0 auxiliary repo should be created
    as /etc/yum.repos.d/7.0_auxiliary

    This helps with upgrade/rollback process for
    Fuel Master.

    The configuration for auxiliary repository on
    deployed Fuel nodes is unchanged.

    Change-Id: Iec2645ddbee6b1b6b9e5c2816707b275542ef51b
    Partial-Bug: #1480282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/211589
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f3cf71170d12852fd8d89465443f6e708ae529ca
Submitter: Jenkins
Branch: master

commit f3cf71170d12852fd8d89465443f6e708ae529ca
Author: Igor Kalnitsky <email address hidden>
Date: Tue Aug 11 17:02:08 2015 +0300

    Remove new auxiliary repo in case of rollback

    Since we're managing auxiliary repo in puppet manifests (creating it)
    and we don't have any sort of rollback puppet manifest, we need to
    clean-up its artifacts in fuel_upgrade script.

    This patch removed all newly created auxiliary repo in case of
    rollback.

    Depends-On: Iec2645ddbee6b1b6b9e5c2816707b275542ef51b
    Closes-Bug: #1480282

    Change-Id: If34406dd77aed45f19fa2f352bd23a0302716053
    Signed-off-by: Igor Kalnitsky <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Artem Hrechanychenko (agrechanichenko)
tags: added: on-verification
tags: removed: on-verification
Artem Hrechanychenko (agrechanichenko)
tags: added: on-verification
Artem Hrechanychenko (agrechanichenko)
tags: removed: on-verification
Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 7.0-301

https://mirantis.testrail.com/index.php?/tests/view/1883346&group_by=cases:section_id&group_order=asc&group_id=4234

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Nick Edwards (njwedwards-3) wrote :
Download full text (4.5 KiB)

Still not working when trying to upgrade from 6.1 to 7.0

2015-10-02 11:46:03 DEBUG 9954 (utils) Execute command "yum install -v -y fuel-7.0.0"
2015-10-02 11:46:03 DEBUG 9954 (utils) Stdout and stderr of command "yum install -v -y fuel-7.0.0":
2015-10-02 11:46:04 DEBUG 9954 (utils) Loading "fastestmirror" plugin
2015-10-02 11:46:04 DEBUG 9954 (utils) Loading "priorities" plugin
2015-10-02 11:46:04 DEBUG 9954 (utils) Not loading "product-id" plugin, as it is disabled
2015-10-02 11:46:04 DEBUG 9954 (utils) Not loading "subscription-manager" plugin, as it is disabled
2015-10-02 11:46:04 DEBUG 9954 (utils) Config time: 0.014
2015-10-02 11:46:04 DEBUG 9954 (utils) Yum Version: 3.2.29
2015-10-02 11:46:04 DEBUG 9954 (utils) Setting up Package Sacks
2015-10-02 11:46:04 DEBUG 9954 (utils) Determining fastest mirrors
2015-10-02 11:46:04 DEBUG 9954 (utils) file:///var/www/nailgun/2014.2.2-6.1/centos/auxiliary/repodata/repomd.xml: [Errno 14] Could not open/read file:///var/www/nailgun/2014.2.2-6.1/centos/auxiliary/repodata/repomd.xml
2015-10-02 11:46:04 DEBUG 9954 (utils) Trying other mirror.
2015-10-02 11:46:04 DEBUG 9954 (utils) Error: Cannot retrieve repository metadata (repomd.xml) for repository: auxiliary. Please verify its path and try again
2015-10-02 11:46:04 ERROR 9954 (upgrade) HostSystemUpgrader: failed to upgrade: "Shell command executed with "1" exit code: yum install -v -y fuel-7.0.0 "
Traceback (most recent call last):
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/upgrade.py", line 82, in run
    upgrader.upgrade()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 97, in upgrade
    self.install_packages()
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/engines/host_system.py", line 143, in install_packages
    utils.exec_cmd('yum install -v -y {0}'.format(package))
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 61, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/var/tmp/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 112, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "1" exit code: yum install -v -y fuel-7.0.0
2015-10-02 11:46:05 DEBUG 9954 (upgrade) Run rollback
2015-10-02 11:46:05 DEBUG 9954 (upgrade) HostSystemUpgrader: rollbacking...
2015-10-02 11:46:05 DEBUG 9954 (utils) Remove file "/etc/yum.repos.d/7.0_nailgun.repo"
2015-10-02 11:46:05 DEBUG 9954 (utils) Removing "/var/www/nailgun/2014.2.2-6.1"
2015-10-02 11:46:05 DEBUG 9954 (utils) Removing "/var/www/nailgun/2015.1.0-7.0"
2015-10-02 11:46:05 INFO 9954 (supervisor_client) Start all services
2015-10-02 11:46:28 INFO 9954 (version_file) Switch current version file to previous version
2015-10-02 11:46:28 DEBUG 9954 (utils) Symlinking "/etc/fuel/6.1/version.yaml" -> "/etc/fuel/version.yaml" [overwrite=1]
2015-10-02 11:46:28 DEBUG 9954 (utils) Removing "/etc/fuel/version.yaml"
2015-10-02 11:46:28 ERROR 9954 (upgrade) *** UPGRADE FAILED
2015-10-02 11:46:28 ER...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-upgrade (master)

Fix proposed to branch: master
Review: https://review.openstack.org/240209

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-upgrade (master)

Reviewed: https://review.openstack.org/240209
Committed: https://git.openstack.org/cgit/openstack/fuel-upgrade/commit/?id=e6dae705a85f2b30f18b2020533afd5a571216a9
Submitter: Jenkins
Branch: master

commit e6dae705a85f2b30f18b2020533afd5a571216a9
Author: Igor Kalnitsky <email address hidden>
Date: Tue Aug 11 17:02:08 2015 +0300

    Remove new auxiliary repo in case of rollback

    Since we're managing auxiliary repo in puppet manifests (creating it)
    and we don't have any sort of rollback puppet manifest, we need to
    clean-up its artifacts in fuel_upgrade script.

    This patch removed all newly created auxiliary repo in case of
    rollback.

    Depends-On: Iec2645ddbee6b1b6b9e5c2816707b275542ef51b
    Closes-Bug: #1480282

    Change-Id: If34406dd77aed45f19fa2f352bd23a0302716053
    Signed-off-by: Igor Kalnitsky <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.