Building of Ubuntu bootstrap image fails: 'Stderr: 'Unexpected error\n404 Client Error: Not Found\n'

[Artem Panchenko]

Fuel version info (8.0 build #281): http://paste.openstack.org/show/481495/

Build verification tests fail, because slave nodes do not discover, Ubuntu image build for bootstrap failed:

Traceback (most recent call last):
  File "/bin/fuel-bootstrap", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/fuel_bootstrap/main.py", line 54, in main
    return fuel_bootstrap_app.run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/fuel_bootstrap/commands/build.py", line 185, in take_action
    data=vars(parsed_args))
  File "/usr/lib/python2.7/site-packages/fuel_bootstrap/utils/bootstrap_image.py", line 191, in call_wrapped_method
    return wrapped_methods[name](**kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_bootstrap/utils/bootstrap_image.py", line 155, in make_bootstrap
    utils.execute(*opts)
  File "/usr/lib/python2.7/site-packages/fuel_agent/utils/utils.py", line 133, in execute
    stderr=stderr, cmd=command)
fuel_agent.errors.ProcessExecutionError: Unexpected error while running command.
Command: fa_mkbootstrap --nouse-syslog --data_driver bootstrap_build_image --nodebug -v --input_data_file /tmp/tmpTjRgSi
Exit code: 255
Stdout: ''
Stderr: 'Unexpected error\n404 Client Error: Not Found\n'

Looks like there was a temporary issue with one of used Ubuntu mirrors:

[root@nailgun ~]# fgrep uri: /etc/fuel-bootstrap-cli/fuel_bootstrap_cli.yaml
      uri: "http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/"
      uri: "http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/"
      uri: "http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/"
      uri: "http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/"
      uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0"
      uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0"
      uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0"
      uri: "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0"
      uri: "http://archive.ubuntu.com/ubuntu"

Image was successfully built manually after the test, so this bug is floating. Fuel bootstrap logging should be improved (it could print failed URL for example) in order to investigate such issues in the future. Also we need to check that bootstrap image is successfully built during Fuel installation in system tests.

[Aleksey Zvyagintsev (azvyagintsev)]

Looks like infra repo problmes:

Stderr: 'E: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/c/ceph/librbd1_0.94.5-0u~u14.04+mos1_amd64.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/p/python-setuptools/python-pkg-resources_18.3.1-2~u14.04+mos1_all.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/p/python-setuptools/python-setuptools_18.3.1-2~u14.04+mos1_all.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/c/cmd2/python-cmd2_0.6.7-2~u14.04+mos1_all.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/p/python-funcsigs/python-funcsigs_0.4-1~u14.04+mos1_all.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/pool/main/s/six/python-six_1.9.0-5~u14.04+mos1_all.deb 404 Not Found [IP: 208.78.244.194 80]\n\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/255940

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksey Zvyagintsev (<email address hidden>) on branch: master
Review: https://review.openstack.org/255940
Reason: moved to discussion
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082001.html

[Sergey Kulanov (skulanov)]

I propose to close this issue after implementing https://bugs.launchpad.net/fuel/+bug/1525133

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257054
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=9f4c5d84873272c3f43e7974c6e772159e10f1ba
Submitter: Jenkins
Branch: master

commit 9f4c5d84873272c3f43e7974c6e772159e10f1ba
Author: Artem Panchenko <email address hidden>
Date: Sun Dec 13 18:16:12 2015 +0200

    Check bootstrap image after Fuel installation

    Add check in tests that bootstrap image is
    successfully built and activated, so slave nodes
    could be booted via PXE.

    Related-bug: #1524855
    Change-Id: I88a8be8f0c906780711438ea6667aa2051220008

[Artem Panchenko (apanchenko-8)]

This issue wasn't fixed (see https://bugs.launchpad.net/fuel/+bug/1527500/comments/4 ), I just contributed related patch to the tests in order to make the logs more informative.

[Aleksey Zvyagintsev (azvyagintsev)]

Guys, please don't change status of this issue -
correct one - won't fix by Fuel build team.
All others- duplicates of this one.

@Artem - link which you provide - don't made any fixes. This issue are floating and appears from time-to time- when builders try's to load package which was removed by build time on mirrors

[Artem Panchenko (apanchenko-8)]

@Aleksey,

> link which you provide - don't made any fixes.

what link are you talking about?

> This issue are floating and appears from time-to time- when builders try's to load package which was removed by build time on mirrors

If the mirrors are "flapping" then we can't use them for tests and jobs should be switched to use persistent repositories snapshots.

We can't just mark this issue as 'won't fix', because it blocks different system tests.

[Aleksandra Fedorova (bookwar)]

Please note, that infra is getting ready for SCF thus jenkins configs are freezed till after SCF - which is this Friday.

If you need patch to 8.0 configs you'lle need to send them to master(9.0) first, and then backport to 8.0

[Roman Vyalov (r0mikiam)]

Reassigned back on Aleksey Zvyagintsev (azvyagintsev) because he prepared the fix https://review.fuel-infra.org/#/c/15010/2

[Sergey Shevorakov (sshevorakov)]

Tag swarm-blocker is set due to this bug fails 3 test cases.

[Aleksey Zvyagintsev (azvyagintsev)]

Partial patch:
https://review.fuel-infra.org/#/c/15010
CI-team please check this patch on fuel-ci-sandbox\whatever.

[Vladimir Kozhukalov (kozhukalov)]

The issue seems fixed because currently bootstrap script uses local mos repo by default.

See these patches

https://review.openstack.org/#/c/258008/
https://review.openstack.org/#/c/261463/

[Vladimir Kozhukalov (kozhukalov)]

As far as we put mos repo on the ISO and then on the Fuel master node we should run tests against this local repo not against online mos repo. The root cause of the issue is that by default we used online mos repo when building bootstrap image. This online repo (if not de-referenced into snapshot url) changes frequently which makes it likely to encounter 404 error during package installing.

These two patches [1] and [2] make it possible to define mos mos-updates mos-security fully independently: local mos, online mos-updates and mos-security.

[1] https://review.openstack.org/#/c/258008/
[2] https://review.openstack.org/#/c/261463/

If we de-reference mos repo into snapshot and then substitute the default one (which is local) like suggested here [3], we devalue the whole test process, because we run tests against repo that is not an exact copy of the local one.

[3] https://review.fuel-infra.org/#/c/15010

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Aleksey Zvyagintsev <email address hidden> on branch: master
Review: https://review.fuel-infra.org/15010

[Roman Podoliaka (rpodolyaka)]

This can still be reproduced:

https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.ironic_deploy_swift/3/

http://paste.openstack.org/show/485271/

[Aleksey Zvyagintsev (azvyagintsev)]

This issue not related to feature-ubuntu-bootstrap - so im removing tag.

[Dmitry Kaigarodеsev (dkaiharodsev)]

@fuel-buld team please check

[Andrey Maximov (maximov)]

removing area-python bug because this is QA/Infra related

[Roman Vyalov (r0mikiam)]

build team cannot fix this problem before HCF. We need to use snapshot of repositories in our swarm/bvt tests. reassign to QA team.

[Tatyanka (tatyana-leontovich)]

Roman I believe you are talking about mos repositories, if so it kinda feature request for qa team and covered by https://bugs.launchpad.net/fuel/+bug/1540476

[Tatyanka (tatyana-leontovich)]

so move for 8.0 to won’t fix

[Roman Vyalov (r0mikiam)]

@Tatyanka: yes, I am talking about mos-repos.

[Oleksandr Minakov (oleksandr-minakov)]

Looks like I can still reproduce this issue https://ci.fuel-infra.org/job/master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy/150/console

http://paste.openstack.org/show/489832/

[Tatyanka (tatyana-leontovich)]

Mark as duplicate of 1540476 according can be caused mirrors update during deploy/test, so usage of snapshots can helps here

[Nastya Urlapova (aurlapova)]

I cannot agree here, that issue is duplicate of bug #1540476. Bootstrap cannot work with other repos only with defaults.

[Ihor Kalnytskyi (ikalnytskyi)]

Nastya,

Why do you think so? There's a fuel-bootstrap configuration file located at

  /etc/fuel-bootstrap-cli/fuel_bootstrap_cli.yaml

and with the content [1]. As you can see you can configure repos on your own in that config. Therefore, you can put OpenStack snapshots here [2].

[1]: http://paste.openstack.org/raw/491427/
[2]: http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-03-22-101501/

[Alexandr Kostrikov (akostrikov-mirantis)]

There were no reproduces since merge of the fix.