Ubuntu bootstrap image wasn't built and activated test failed

[Ilya Kutukov]

Expected result - test is passed

Actual result -

https://ci.fuel-infra.org/job/master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy/494/console

<<<<<****************************************************************************************************>>>>>
2016-03-14 19:01:02,251 - INFO decorators.py:299 -- Failed in method setup_master. Make snapshot: error_setup_master
2016-03-14 19:01:02,251 - INFO decorators.py:309 -- You could revert and ssh to master node: [source /home/jenkins/venv-nailgun-tests-2.9/bin/activate; dos.py revert-resume env_master_review_fuel_web_deploy error_setup_master && ssh root@10.109.5.2]
2016-03-14 19:01:02,251 - INFO decorators.py:311 -- <<<<<****************************************************************************************************>>>>>
2016-03-14 19:01:02,252 - ERROR decorators.py:111 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/systest/master/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/systest/master/fuelweb_test/tests/base_test_case.py", line 167, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/systest/master/fuelweb_test/models/environment.py", line 470, in setup_environment
    self.wait_bootstrap()
  File "/home/jenkins/workspace/systest/master/fuelweb_test/models/environment.py", line 628, in wait_bootstrap
    self.bootstrap_image_check()
  File "/home/jenkins/workspace/systest/master/fuelweb_test/models/environment.py", line 659, in bootstrap_image_check
    'Ubuntu bootstrap image wasn\'t built and activated! '
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Ubuntu bootstrap image wasn't built and activated! See logs in /var/log/fuel-bootstrap-image-build.log for details.

[Ilya Kutukov (ikutukov)]

Test is failing accidentally

[Dmitry Pyzhov (dpyzhov)]

Looks like broken test. Please attach diagnostic snapshot and return to the python team if you see issues in the product.

[Artem Panchenko (apanchenko-8)]

Dmitry,

I don't think the tests are broken, here is a part of fuel-bootstrap-image-build.log:

http://paste.openstack.org/show/491114/

Also diagnostic snapshot is available in job artifacts (see link in the description).

[Alexandr Kostrikov (akostrikov-mirantis)]

@Artem Panchenko - why does link points to http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest?
We should use snapshots in tests. Probably, we should update config file for bootstrap image build to point to snapshot?

[Dmitry Pyzhov (dpyzhov)]

Build failed because of missed package in the mirror. How does it related to python?

[Artem Panchenko (apanchenko-8)]

@Aleksandr,

AFAIK ubuntu-latest is a link to the latest snapshot (currently seems it points to ubuntu-2016-03-17-170029)

@Dmitry.

Agree, but how does it related to tests? Assigning this bug to fuel-build team.

[Sergey Kulanov (skulanov)]

@Folks,

I propose to switch to snapshot usage, we need to ask CI-team to discuss and make decision which mirrors to use:
and probably re-define this variable MIRROR_UBUNTU

++ curl -sSf https://ci.fuel-infra.org/job/devops.master.env/lastSuccessfulBuild/artifact/mirror_ubuntu_data.txt
+ export 'MIRROR_UBUNTU=deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty main universe multiverse|deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty-updates main universe multiverse|deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty-security main universe multiverse'
+ MIRROR_UBUNTU='deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty main universe multiverse|deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty-updates main universe multiverse|deb http://mirror.seed-cz1.fuel-infra.org/pkgs//ubuntu-latest/ trusty-security main universe multiverse'

[Aleksandra Fedorova (bookwar)]

env updated

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/18210
Submitter: Alexander Evseev <email address hidden>
Branch: master

Commit: 37b2db9367d28acf16a628f96f876cdcde1d1aea
Author: Alexander Evseev <email address hidden>
Date: Mon Mar 21 10:42:45 2016

[pkg] Fix system test workflow

  * Define distro name for Ubuntu and directory name for CentOS repo
    using JJB values repo_name_sep and repo_suff_rpm.
  * Do not use unneeded anymore artifact buildresult.params.
  * Fix setting repo paths in build job by using hardcoded OS names.
  * Modify get_deb_snapshot() to keep original distro name and component(s).
  * Modify mirror choosing to get host for any use.
  * Remove setting of PERESTROIKA_REPO.
  * Do not add "release" repository to extra RPM repos list.
  * Always setup Ubuntu cluster nodes (OPENSTACK_RELEASE).
  * Remove stale publisher artifact at job start.
  * Create one more artifact containing parameters for systest.
  * Use above artifact to set parameters in systest job.
  * If parameter UPDATE_FUEL is set, then also set parameters CUSTOM_ENV
    and UPDATE_MASTER to true, and UPDATE_FUEL_MIRROR to CR repository if job
    has parameter RPM_REPO_URL or to mirrored "release" repository snapshot
    otherwise.

Change-Id: Icfab1e541bd40e4f3c67285dae2e98821f6d473d
Closes-Bug: #1558571
Closes-Bug: #1557488

[Bogdan Dobrelya (bogdando)]

This still blocks CI for the master (10.0)

[Dmitry Mescheryakov (dmitrymex)]

The issue reproduced again on 9.0:

https://packaging-ci.infra.mirantis.net/job/9.0-pkg-systest-ubuntu/742/

The symptoms are the same, in console log:
AssertionError: Ubuntu bootstrap image wasn't built and activated! See logs in /var/log/fuel-bootstrap-image-build.log for details.

In fuel-bootstrap-image-build.log in the snapshot the relevant line is:
got an error: 404 Client Error: Not Found for url: http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/dists/mos9.0-security/Release

[Aleksandra Fedorova (bookwar)]

this is not blocking issue, moving to High

[Artur Kaszuba (akaszuba)]

This problem is caused by bootstrap which is internal part of fuel. We use snapshoted mirrors in tests and probably it is not respected by bootstrap part. This could create problems in some situations.
Could you change fuel-qa/devops code to respect snapshoted mirrors by bootstrap?