Random problems fetching packages during Fuel 8 build

Bug #1536538 reported by Stefan K. Berg on 2016-01-21
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Fuel build team

Bug Description

When building Fuel@OPNFV, we're frequently getting build errors as packages from the fuel-infra mirrors fails to download.
The current workaround is to gradually build up the package cache (iteratively doing "make clean; make iso") but this
is no long-term solution.

Right now I'm using the "cz" mirror which seemed slightly better than the "ext" one, but we're very seldom able to build
cleanly straight away.

I'm thinking bandwidth issues upstream, or DNS load balancing and not all replicas at sync - but that's just a hunch.

The entire Fuel@OPNFV community (with a release deadline coming up in two weeks) would be super grateful if someone
could look into this!

Cheers,

Stefan

Michal Skalski (mskalski) wrote :

Example log of failed build can be found here: https://build.opnfv.org/ci/job/fuel-verify-build-master/458/console

tags: added: area-build
Changed in fuel:
importance: Undecided → High
Changed in fuel:
assignee: nobody → Fuel build team (fuel-build)
milestone: none → 8.0
Sergey Kulanov (skulanov) wrote :

@Folks, you need to use snapshot approach since MOS mirrors updating each ~15 minutes,

This this how it's implemented in our job [1], in short:

1. extract info from http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/os.target.txt
       snapshots/os-2016-01-21-093001
2. Construct new path:
   http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/snapshots/os-2016-01-21-093001/
 instead of
   http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/os/

the same is true for ubuntu:
1. http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0.target.txt
     snapshots/8.0-2016-01-21-093001
2. new path:
     http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/8.0-2016-01-21-093001/
 instead of
    http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/

[1]. https://github.com/fuel-infra/jenkins-jobs/blob/master/servers/jenkins-product-ci/8.0/builders/custom.sh#L46-L95

ps. setting to incomplete, please feel free to update bug if you have further questions

Changed in fuel:
status: New → Incomplete
Michal Skalski (mskalski) wrote :

We already tested build with using snapshots for mos-ubuntu repositories which does not improve our situations. I will add snapshot mechanism also for mos-centos and will come back with results.

Roman Vyalov (r0mikiam) on 2016-01-26
Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers