mariadb mirror impossibly slow

[Jesse Pretorius]

It appears that due to a very slow mariadb mirror today, it takes ~30 mins per container/host to complete the basic addition of the repo to the apt sources and to update the cache.

We either need to implement some sort of mechanism to provide alternative mirrors for mariadb (as the mariadb repositories have often been build blockers), and/or we need to implement a shorter timeout (it seems that the default is 5 mins) for apt so that it fails faster if there's a connection issue.

reference for timeout config: http://askubuntu.com/questions/141513/how-to-lower-wait-time-for-repository-updates

reference for mirror method: http://askubuntu.com/questions/39922/how-do-you-select-the-fastest-mirror-from-the-command-line/9035#9035

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/148324

[Jesse Pretorius (jesse-pretorius)]

I think the ideal way to resolve this would be to implement the mirror method for the apt source. Unfortunately the mirror method only supports fetching the mirror list via http (it does not support using a local file list). I've logged a bug upstream to include that feature: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1412758

[Jesse Pretorius (jesse-pretorius)]

FYI the timeout method was tested with https://review.openstack.org/148324 patch set 2 and did not work. It appears that the repository in question does allow a connection (it does not refuse it), so the timeout configuration does not apply.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: icehouse
Review: https://review.openstack.org/148905

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/148906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148324
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=c6bee47bc92de988cb366e23a49255485df21834
Submitter: Jenkins
Branch: master

commit c6bee47bc92de988cb366e23a49255485df21834
Author: Jesse Pretorius <email address hidden>
Date: Mon Jan 19 17:47:38 2015 +0000

    Use reliable static mirror for mariadb apt repository

    This patch enables the use of the rackspace mirror of the mariadb
    archive. This is necessary as archive.mariadb.org is not always
    responding appropriately to apt requests.

    Change-Id: I9005c6f03fd88345028183f38c682c7734562b24
    Closes-Bug: #1412521

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: juno
Review: https://review.openstack.org/148906
Reason: This patch can't be added individually in this branch - it needs to be squashed with other patches which combined together get gating working.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: icehouse
Review: https://review.openstack.org/148905
Reason: This patch can't be added individually in this branch - it needs to be squashed with other patches which combined together get gating working.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/149230

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: juno
Review: https://review.openstack.org/149230
Reason: I made a mistake with the commit message.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149209
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=e01165bcc729ba151473dfe3199f913906e8d4b9
Submitter: Jenkins
Branch: juno

commit e01165bcc729ba151473dfe3199f913906e8d4b9
Author: Jesse Pretorius <email address hidden>
Date: Mon Jan 19 17:47:38 2015 +0000

    Squash all patches required to make gating work

    This is a merge of the following patches:
    62d2525 Fixed generator type assumption error
    1339640 Allow swift_vars to not be specified for swift_hosts
    735eb68 AIO to use https to access the frozen repo
    033df65 use archive.mariadb.org for to install mariadb package
    8b9cf6e Use reliable static mirror for mariadb apt repository
    b30f9d1 Add debootstrap package to LXC package list
    5c464e1 Add swift_all inventory to swift-common play
    a88bcb7 Add swift_all inventory to swift account/container/object plays

    Change-Id: I630e38b35a00fe547ec3aaa68604a5902be4f83e
    Closes-Bug: #1399387
    Closes-Bug: #1406323
    Closes-Bug: #1399387
    Closes-Bug: #1402582
    Closes-Bug: #1402594
    Closes-Bug: #1402617
    Closes-Bug: #1409762
    Closes-Bug: #1410743
    Closes-Bug: #1411331
    Closes-Bug: #1412521

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: icehouse
Review: https://review.openstack.org/149213
Reason: This patch was an investigation to figure out what needed to go into the branch in order to get the gate operational. Now that we know, we can cherry-pick the patches individually.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148905
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=4d7096d6434d35936c620579cfee2b88632afa04
Submitter: Jenkins
Branch: icehouse

commit 4d7096d6434d35936c620579cfee2b88632afa04
Author: Jesse Pretorius <email address hidden>
Date: Mon Jan 19 17:47:38 2015 +0000

    Use reliable static mirror for mariadb apt repository

    This patch enables the use of the rackspace mirror of the mariadb
    archive. This is necessary as archive.mariadb.org is not always
    responding appropriately to apt requests.

    Change-Id: I9005c6f03fd88345028183f38c682c7734562b24
    Closes-Bug: #1412521
    Related-Bug: #1409762

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: juno
Review: https://review.openstack.org/149230
Reason: mistaken commit

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148906
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=48b25c6f4c9335085e9ae57f5d764c3b0fee8b0c
Submitter: Jenkins
Branch: juno

commit 48b25c6f4c9335085e9ae57f5d764c3b0fee8b0c
Author: Jesse Pretorius <email address hidden>
Date: Mon Jan 19 17:47:38 2015 +0000

    Use reliable static mirror for mariadb apt repository

    This patch enables the use of the rackspace mirror of the mariadb
    archive. This is necessary as archive.mariadb.org is not always
    responding appropriately to apt requests.

    Closes-Bug: #1412521
    Change-Id: I9005c6f03fd88345028183f38c682c7734562b24
    (cherry picked from commit c6bee47bc92de988cb366e23a49255485df21834)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jesse Pretorius (<email address hidden>) on branch: juno
Review: https://review.openstack.org/149230
Reason: never mind - borked