Deployment failed on connectivity check to http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/ after stop deployment action

[Tatyanka]

Steps:
Scenario:
1. Create cluster in HA mode with 1 controller
2. Add 1 node with controller role
3. Add 1 node with compute role
4. Run provisioning task
5. Stop provisioning
6. Reset settings
7. Add 1 node with cinder role
8. Re-deploy cluster
9. Run OSTF

Actual Result:
Re-deploy cluster failed with
2015-09-08 03:31:50,229 - INFO fuel_web_client.py:1194 -- Task finished. Took 30 seconds. {
 "status": "error",
 "name": "deploy",
 "cluster": 1,
 "result": {},
 "progress": 100,
 "message": "Connection to following repositories could not be established: http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/",
 "id": 4,
 "uuid": "a317a070-e0cc-4f07-b1e5-6a6dd47aa6a6"

    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "287"
  build_id: "287"
  nailgun_sha: "46a7a2177a0b7ef91422284c1c90295fee8f5c84"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "43224223dab8cf9627b5ecf737e60216a3fdd114"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

[Ihor Kalnytskyi (ikalnytskyi)]

Today the "stop_reset_on_provisioning" test has been passed successfully. Still, the error has been reproduced on other test.
I'm unable to find anything useful in logs, except the fact that sometime the CheckBeforeDeployment task fails due to connectivity issue. If I revert snapshot and run it manually - it passes.

So it looks like it's indeed a connectivity issue, perhaps our mirrors are weak to handle a lot of requests.

[Ihor Kalnytskyi (ikalnytskyi)]

I think the best what we can do is to add both retries and information about HTTP error.

[Dennis Dmitriev (ddmitriev)]

Confirmed issue with repositories.

Reproduced in system tests *after* a deploy was successfully finished.

When 'apt-get update' was executed on one of the nodes, webserver where repository is located suddenly returned HTTP error 404.

Time when it was happened: 12:30 UTC.

Looks like some hidden actions are performed on the mirrors with repositories, and this lead to unavailability or inconsistency of repositories from time to time.

================================
DevopsCalledProcessError: Command 'apt-get clean all && apt-get update > /dev/null' returned non-zero exit status 100
W: GPG error: http://mirror.fuel-infra.org mos7.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CA2B20483E301371

W: GPG error: http://mirror.fuel-infra.org mos7.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CA2B20483E301371

W: GPG error: http://mirror.fuel-infra.org mos7.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CA2B20483E301371

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/main/binary-amd64/Packages 404 Not Found [IP: 208.78.244.194 80]

W: Failed to fetch http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/dists/mos7.0-updates/restricted/binary-amd64/Packages 404 Not Found [IP: 208.78.244.194 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.
==========================

[Igor Shishkin (teran)]

Looks pretty similar to what we had about mirror syncs: we've tried to fetch something before mirror was synced.

[Andrew Woodward (xarses)]

Patch is https://review.openstack.org/#/c/222172/ but is linked to wrong bug id

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222172
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f040fc85be61d79d98343d1c3def0eef4393e54f
Submitter: Jenkins
Branch: master

commit f040fc85be61d79d98343d1c3def0eef4393e54f
Author: Igor Kalnitsky <email address hidden>
Date: Thu Sep 10 15:35:34 2015 +0300

    Add retries to repo connectivity checker

    Sometimes under heavy load, infra may return 5xx errors. It's ok, and
    most package installer are able to handle it properly. So let's
    introduce retries to check connectivity, so our system tests will stop
    fail time-to-time.

    Related-Bug: #1493390

    Change-Id: I96c2c2b6e94428085d47c2d40e0ae3753304a8f6
    Signed-off-by: Igor Kalnitsky <email address hidden>

[Igor Shishkin (teran)]

Removing 7.0 milestone since the task is not related there anymore.

[Sergey Kulanov (skulanov)]

Folks this happened because there was switch snapshots in mirrors.
Let's re-open in case further issues, because in this case we have two ways:
1. Always dereference http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0.target.txt, like we do on ISO build
2. Makes mirror updates less often