openstack-infra jobs timing out due to slow connectivity to review.fuel-infra.org

[Alex Schultz]

There appears to be connectivity issues. We're getting jobs taking longer than an hour to complete and failing (normal time is < 10 mins). Additionally we're getting timeouts when pulling down modules.

See the following reviews & their job logs:

https://review.openstack.org/#/c/268195/
http://logs.openstack.org/95/268195/5/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/ab27488/

https://review.openstack.org/#/c/269507/
http://logs.openstack.org/07/269507/3/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/423dab0/

https://review.openstack.org/#/c/269110/
http://logs.openstack.org/10/269110/2/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/1c5e327/

https://review.openstack.org/#/c/268214/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/4315ea9/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/efc8667/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/bbf7208/

https://review.openstack.org/#/c/267483/
http://logs.openstack.org/83/267483/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/40a967e/
http://logs.openstack.org/83/267483/2/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/327982a/

https://review.openstack.org/#/c/269207/
http://logs.openstack.org/07/269207/2/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e3ed5b5/

https://review.openstack.org/#/c/268147/
http://logs.openstack.org/47/268147/7/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/7a974d6/

https://review.openstack.org/#/c/269564/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/1932871/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/9f38565/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/d727dbe/

https://review.openstack.org/#/c/267569/
http://logs.openstack.org/69/267569/4/gate/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/eb37579/

https://review.openstack.org/#/c/269522/
http://logs.openstack.org/22/269522/1/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/4f1b729/

https://review.openstack.org/#/c/266512/
http://logs.openstack.org/12/266512/7/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/72b7def/
http://logs.openstack.org/12/266512/7/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/0131a4d/

https://review.openstack.org/#/c/268082/
http://logs.openstack.org/82/268082/5/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/316cb01/

https://review.openstack.org/#/c/267989/
http://logs.openstack.org/89/267989/4/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/ae06fa8/

https://review.openstack.org/#/c/269694/
http://logs.openstack.org/94/269694/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/4d2d2b2/

https://review.openstack.org/#/c/262587/
http://logs.openstack.org/87/262587/3/gate/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e1f02c0/
http://logs.openstack.org/87/262587/3/gate/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/8e8a919/

https://review.openstack.org/#/c/263807/
http://logs.openstack.org/07/263807/2/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/6ee2e34/
http://logs.openstack.org/07/263807/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/9685b24/
http://logs.openstack.org/07/263807/2/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/146239f/
http://logs.openstack.org/07/263807/2/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/b27dae2/

https://review.openstack.org/#/c/268133/
http://logs.openstack.org/33/268133/2/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/5be5781/
http://logs.openstack.org/33/268133/2/gate/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/5be6d33/

https://review.openstack.org/#/c/262681/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/6370b0b/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/2256497/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/26503d8/

https://review.openstack.org/#/c/265945/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/c7239a7/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/33ee67d/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/dca607a/

https://review.openstack.org/#/c/269527/
http://logs.openstack.org/27/269527/2/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/d40c636/

https://review.openstack.org/#/c/269749/
http://logs.openstack.org/49/269749/3/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/a372690/
http://logs.openstack.org/49/269749/3/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e3aa520/

https://review.openstack.org/#/c/263426/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/30669a9/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/f414290/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/0dfda41/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/c803b3a/

https://review.openstack.org/#/c/267623/
http://logs.openstack.org/23/267623/4/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/978c733/

https://review.openstack.org/#/c/269585/
http://logs.openstack.org/85/269585/2/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/d2befb7/
http://logs.openstack.org/85/269585/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/68d3eca/

https://review.openstack.org/#/c/269607/
http://logs.openstack.org/07/269607/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/e4941ea/
http://logs.openstack.org/07/269607/1/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/0aa84e8/

https://review.openstack.org/#/c/269676/
http://logs.openstack.org/76/269676/2/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/5d729b0/
http://logs.openstack.org/76/269676/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/d9c2e1e/

[Igor Shishkin (teran)]

CI team, please enable dmesg gathering the same was as we're getting those logs.
As I got from OpenStack Infra folks there's a way to enable it in job configuration.
This will really help to get the real reason of those failure.

For now I have two suspicions:
- instance running our test is get out of some limits(should be reflected in dmesg)
- our Gerrit is out of it's RPS capacity(no clear sings of it on graphs, but it's possible)

Having dmesg logging will make us possible to get it.

Reassigning to CI team to enable logging.

[Igor Belikov (ibelikov)]

Igor, I created a separate bug for log gathering part: https://bugs.launchpad.net/fuel/+bug/1536160

[Aleksandra Fedorova (bookwar)]

Please proceed with git replication to GitHub workaround.

[Igor Shishkin (teran)]

@Alexandra, we're researching what would be the fastest way to achieve to dedication those repos to github or dedicated VM whatever is faster.

[Andrey Nikitin (heos)]

We've added those repositories on Github and enabled replication on them from Fuel Gerrit. The first replication of the projects on github works slowly.

List of projects are replication on Github: https://github.com/fuel-infra.

So, I think we can use them as workaround or to find a root cause of the problem.

[Andrey Nikitin (heos)]

I'm going to reduce Importance due to we have workaround for the problem.

[Andrey Nikitin (heos)]

Fuel CI team, please take care about the bug, when replication will finished.

[Andrey Nikitin (heos)]

Replication to Github is finished, please replace review.fuel-infra.org on those jobs.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/270277

[Alex Schultz (alex-schultz)]

Tags were not propagated, please fix so we can switch them out.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/270313

[Andrey Nikitin (heos)]

All task of the projects were replicated too. Please check.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Andrey Nikitin <email address hidden>
Review: https://review.fuel-infra.org/16341

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16341
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 1e6e8bdbdfa864cb40f166da59c9bb1f2afb3061
Author: Andrey Nikitin <email address hidden>
Date: Thu Jan 21 13:25:36 2016

Added 'read' permission to fuel-infra-github group

This permissions is needed to allow replication the projects
on github.

Change-Id: I4a85eb80aa779ad103e25e980c1b195d20f16f0b
Related-bug: #1535832

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270313
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=abd991b2305fd0ba52c29211feff3325439489bb
Submitter: Jenkins
Branch: master

commit abd991b2305fd0ba52c29211feff3325439489bb
Author: Alex Schultz <email address hidden>
Date: Wed Jan 20 09:42:55 2016 -0700

    Update CI tasks and module fixtures

    This change updates the fixtures files for the modules within
    fuel-library to use the relative path to the upstream modules used by
    fuel-library. With this change we are also adding an initial prep task
    that is used by CI to pull down all the modules first and only one time
    to reduce the amount of repeated calls to the upstream git repositories.

    Included in this change is also an update to exclude all the upstream
    modules from the spec and lint tasks as those are maintained in the
    upstream modules and should not be done when testing fuel-library.

    This change also adds documentation to the README.md on how to run the
    tests from the root of fuel-library.

    Change-Id: I9cde21b94a8367fd45cf046af48d7618076a385c
    Related-Bug: #1535832

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270277
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f26a6b23b08280d0ccdbb9b5252abbd9ba49b1da
Submitter: Jenkins
Branch: master

commit f26a6b23b08280d0ccdbb9b5252abbd9ba49b1da
Author: Igor Belikov <email address hidden>
Date: Wed Jan 20 18:57:22 2016 +0300

    Switch librarian git links to github mirror

    This change also updates the puppetfile validation script to allow for
    github.com/fuel-infra

    Change-Id: I274ec4c2f12b9effb76d59499d58ee713c78664d
    Related-Bug: #1535832
    Co-Authored-By: Alex Schultz <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274712
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0375f153b5e7d960e605e414e630525ce23bc3bb
Submitter: Jenkins
Branch: stable/8.0

commit 0375f153b5e7d960e605e414e630525ce23bc3bb
Author: Alex Schultz <email address hidden>
Date: Wed Jan 20 09:42:55 2016 -0700

    Update CI tasks and module fixtures

    This change updates the fixtures files for the modules within
    fuel-library to use the relative path to the upstream modules used by
    fuel-library. With this change we are also adding an initial prep task
    that is used by CI to pull down all the modules first and only one time
    to reduce the amount of repeated calls to the upstream git repositories.

    Included in this change is also an update to exclude all the upstream
    modules from the spec and lint tasks as those are maintained in the
    upstream modules and should not be done when testing fuel-library.

    This change also adds documentation to the README.md on how to run the
    tests from the root of fuel-library.

    Change-Id: I9cde21b94a8367fd45cf046af48d7618076a385c
    Related-Bug: #1535832