Ubuntu package archive periodically inconsistent causing gate build failures

[Sean Dague]

From time to time the archive.ubuntu.com mirrors are inconsistent, so and apt-get install will fail with a package that doesn't exist (though and apt-get update said that it did).

This results in issues like:

2014-02-26 20:22:58.696 | The following extra packages will be installed:
2014-02-26 20:22:58.698 | postgresql-9.1 postgresql-client-9.1 postgresql-client-common
2014-02-26 20:22:58.700 | postgresql-common ssl-cert
2014-02-26 20:22:58.701 | Suggested packages:
2014-02-26 20:22:58.703 | oidentd ident-server locales-all postgresql-doc-9.1 openssl-blacklist
2014-02-26 20:22:58.705 | The following NEW packages will be installed:
2014-02-26 20:22:58.707 | postgresql postgresql-9.1 postgresql-client-9.1 postgresql-client-common
2014-02-26 20:22:58.708 | postgresql-common ssl-cert
2014-02-26 20:22:58.710 | 0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
2014-02-26 20:22:58.712 | Need to get 5380 kB/5392 kB of archives.
2014-02-26 20:22:58.714 | After this operation, 15.0 MB of additional disk space will be used.
2014-02-26 20:22:58.715 | Get:1 http://archive.ubuntu.com/ubuntu/ precise-updates/main postgresql-client-common all 129ubuntu1 [25.1 kB]
2014-02-26 20:22:58.717 | Err http://archive.ubuntu.com/ubuntu/ precise-updates/main postgresql-client-9.1 amd64 9.1.11-0ubuntu0.12.04
2014-02-26 20:22:58.719 | 404 Not Found [IP: 91.189.91.13 80]
2014-02-26 20:22:58.721 | Get:2 http://archive.ubuntu.com/ubuntu/ precise-updates/main postgresql-common all 129ubuntu1 [96.5 kB]
2014-02-26 20:22:58.723 | Err http://archive.ubuntu.com/ubuntu/ precise-updates/main postgresql-9.1 amd64 9.1.11-0ubuntu0.12.04
2014-02-26 20:22:58.724 | 404 Not Found [IP: 91.189.91.13 80]
2014-02-26 20:22:58.726 | Get:3 http://archive.ubuntu.com/ubuntu/ precise-updates/main postgresql all 9.1+129ubuntu1 [5552 B]
2014-02-26 20:22:58.728 | Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/postgresql-9.1/postgresql-client-9.1_9.1.11-0ubuntu0.12.04_amd64.deb 404 Not Found [IP: 91.189.91.13 80]
2014-02-26 20:22:58.730 | Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/postgresql-9.1/postgresql-9.1_9.1.11-0ubuntu0.12.04_amd64.deb 404 Not Found [IP: 91.189.91.13 80]
2014-02-26 20:22:58.732 | Fetched 127 kB in 0s (161 kB/s)
2014-02-26 20:22:58.734 | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
2014-02-26 20:22:58.735 | ++ failed
2014-02-26 20:22:58.737 | ++ local r=100
2014-02-26 20:22:58.739 | +++ jobs -p
2014-02-26 20:22:58.741 | ++ kill
2014-02-26 20:22:58.743 | stack.sh failed: full log in /opt/stack/new/devstacklog.txt.2014-02-26-202130

Logstash query for this - message:"E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?" AND NOT build_name:check-tripleo* AND (filename:console.html OR filename:"logs/devstacklog.txt")

Note: we explicitly exclude the tripleo builds as they are in a different cloud, not using the same mirrors, so their failures are similar but with a different solution.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/77435

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77435
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=64188df36817fc8e80b39f559ce878b60fb3f2d3
Submitter: Jenkins
Branch: master

commit 64188df36817fc8e80b39f559ce878b60fb3f2d3
Author: Sean Dague <email address hidden>
Date: Sun Mar 2 10:01:19 2014 -0500

    added query for broken rax mirror

    Change-Id: Ie5a154e417c5e6a7568800f795bdce2dbf84b1d8
    Related-Bug: #1286818

[Jeremy Stanley (fungi)]

I'm not really sure what we can/should be doing here, other than to either ignore package list update failures, have someone improve rackspace's package repository mirroring algorithm, or start doing all our own package mirroring (which seems like overkill). Setting to low for now because it seems to happen fairly infrequently.

[Judd OBannon (judd-obannon)]

I'm not seeing any reference to Rackspace's mirrors, mirror.rackspace.com. I only see references to achive.ubuntu.com. Could you please provide output indicating the issue with mirror.rackspace.com or change the title to reflect the issue with achive.ubuntu.com. If there's something I can fix I'd love to, but I can't fix archive.ubuntu.com.

[Jeremy Stanley (fungi)]

Judd: thanks for spotting this! I think it was getting confused with bug 1251117 (which *is* for similar issues we hit on mirror.rackspace.com/ubuntu from time to time. I've corrected the title and description of this bug report accordingly.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/98381

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98381
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=3bab55f657d87ba9e8347a0144109ebbcdde017d
Submitter: Jenkins
Branch: master

commit 3bab55f657d87ba9e8347a0144109ebbcdde017d
Author: Sean Dague <email address hidden>
Date: Fri Jun 6 07:49:23 2014 -0400

    this is the expired version of

    Change-Id: I2e7f907fd54aacec85092e9cb3ed61703ac588e4
    Related-Bug: #1286818

[Angus Lees (gus)]

apt will try multiple repository URLs in order when fetching. So we can add other archive mirrors(*) in some fallback order, presumably ending up with a canonical Canonical server (heh). Presumably we can find _one_ mirror on the planet that is internally consistent.

(*) List of mirrors: https://launchpad.net/ubuntu/+archivemirrors

[Jeremy Stanley (fungi)]

In actuality, apt does *not* support fallback to additional repositories for the same packages, confirmed myself by testing in various constructed failure scenarios. Bug 601916 has been open against apt for several years asking for it to be correctly implemented.

For now I'm experimenting with https://review.openstack.org/107060 in an attempt to implement a hackish fallback mechanism into DevStack itself, but long term the best solution will be maintaining local validated mirrors in each provider.

[Sergey Skripnick (eyerediskin)]

http://logs.openstack.org/24/123424/2/gate/gate-rally-dsvm-rally/5cd78b6/logs/devstacklog.txt.gz

Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libnspr4 amd64 2:4.10.7-0ubuntu0.14.04.1
  404 Not Found
Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libnspr4-dev amd64 2:4.10.7-0ubuntu0.14.04.1
  404 Not Found

[Jeremy Stanley (fungi)]

Sergey, your failures match bug 1251117 instead (same behavior, but different mirrors).

[Sean Dague (sdague)]

This seems to mostly only happen against az3.clouds.archive.ubuntu.com which is baked into the HP cloud images.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178160
Committed: https://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=8bc8d33f260193dd01ce090a1dc3ba168ca4394e
Submitter: Jenkins
Branch: master

commit 8bc8d33f260193dd01ce090a1dc3ba168ca4394e
Author: Monty Taylor <email address hidden>
Date: Tue Apr 28 08:15:46 2015 -0400

    Avoid vendor supplied apt mirrors

    They are nuking the world

    Closes-Bug: #1286818
    Change-Id: Id876c8a0dea500b90b3ede1502bc757bfb71c4d4

[Jeremy Stanley (fungi)]

We fixed this by creating package mirrors with generated indices based on the included packages. Subsequent hits on this bug seem to be due to loose pattern matches including runs with random package download failures unrelated to index mismatches.