Random problems fetching packages during Fuel 8 build

[Stefan Berg]

When building Fuel@OPNFV, we're frequently getting build errors as packages from the fuel-infra mirrors fails to download.
The current workaround is to gradually build up the package cache (iteratively doing "make clean; make iso") but this
is no long-term solution.

Right now I'm using the "cz" mirror which seemed slightly better than the "ext" one, but we're very seldom able to build
cleanly straight away.

I'm thinking bandwidth issues upstream, or DNS load balancing and not all replicas at sync - but that's just a hunch.

The entire Fuel@OPNFV community (with a release deadline coming up in two weeks) would be super grateful if someone
could look into this!

Cheers,

Stefan

[Michal Skalski (mskalski)]

Example log of failed build can be found here: https://build.opnfv.org/ci/job/fuel-verify-build-master/458/console

[Sergey Kulanov (skulanov)]

@Folks, you need to use snapshot approach since MOS mirrors updating each ~15 minutes,

This this how it's implemented in our job [1], in short:

1. extract info from http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/os.target.txt
       snapshots/os-2016-01-21-093001
2. Construct new path:
   http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/snapshots/os-2016-01-21-093001/
 instead of
   http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos7-fuel/os/

the same is true for ubuntu:
1. http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0.target.txt
     snapshots/8.0-2016-01-21-093001
2. new path:
     http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/8.0-2016-01-21-093001/
 instead of
    http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/

[1]. https://github.com/fuel-infra/jenkins-jobs/blob/master/servers/jenkins-product-ci/8.0/builders/custom.sh#L46-L95

ps. setting to incomplete, please feel free to update bug if you have further questions

[Michal Skalski (mskalski)]

We already tested build with using snapshots for mos-ubuntu repositories which does not improve our situations. I will add snapshot mechanism also for mos-centos and will come back with results.