Add excplicit repo connectivity checks for Ubuntu envs

[Vyacheslav Struk]

Ubuntu environments need an internet access to deploy successfully:

1) Admin network (fuel master acts as gateway)
2) Public network ("public gateway" at fuel settings acts as gateway)

We should check both and warn user if there is no internet connectivity via this networks.
OR
Check connection to all the repositories mentioned in fuel-ui.

[Maciej Kwiek (maciej-iai)]

Changes that are in progress:
https://review.openstack.org/#/c/172954/ (Astute change)
https://review.openstack.org/#/c/172953/ (fuel-web change)

The first one adds execute_shell action to astute. It allows nailgun to run arbitrary shell commands on slave nodes. The action in dispatcher.rb uses mcollective agent that was already available and used in Astute internally.

The second one adds new task, which calls execute_shell task in astute to check if wget is able to retrieve Release file from repositories configured in current cluster. If any one of them is unavailable then the task return exit code 1 and the task ends up in error state.
This task is called in ApplyChangesTaskManager.check_before_deployment method. The manager is waiting on a barrier, which is closed after receiverd process gets a response from astute. Then task status is checked and proceeds as before or fails deployment.

TODO: check only ubuntu nodes (centos nodes don't need access to ubuntu repos).
TODO: think about way for deployment to not hang if astute fails to give a response (maybe retry the connectivity check).
TODO: fix tests broken by barrier called in ApplyChangesTaskManager.

[Maciej Kwiek (maciej-iai)]

Status update: most of the tests are fixed, CI should work fine on this change today. Will continue with other todos after that.

[Maciej Kwiek (maciej-iai)]

Status update: https://review.openstack.org/#/c/172954/ should work fine on CI. Still not happy with quality of code, will polish it on monday and seek review.

[Maciej Kwiek (maciej-iai)]

Status update: There are some disagreements in how exactly we should do it, I will investigate and it will probably be done - one way or another - tomorrow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/180969

[Bogdan Dobrelya (bogdando)]

Related bug https://bugs.launchpad.net/fuel/+bug/1261940

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180969
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=c4594fc2461f1cf66e580a07d32a869c3f25678d
Submitter: Jenkins
Branch: master

commit c4594fc2461f1cf66e580a07d32a869c3f25678d
Author: Maciej Kwiek <email address hidden>
Date: Thu May 7 13:18:48 2015 +0200

    Check master node connection to repositories

    Added CheckRepositoryConnectionTask which tries to retrieve files from
    repositories and fails if it cannot do it. This check is performed
    before provisioning takes place, so it covers the IBP need
    for master node to have access to repositories for building an image.

    Above task is mocked for integration tests.

    Fake task mode causes retrieving failed repositories method to be
    replaced by method always returning empty list.

    Change-Id: I1af89833e7de8447e73b9d9f00dcc2d9170ea35f
    Partial-Bug: #1439686

[Mike Scherbakov (mihgen)]

Folks,
the whole title of this issue doesn't sound correct to me. Why do we need to check for Internet? We only need to check connectivity to Ubuntu repos, as far as I know. No need to check for 8.8.8.8 or something like this. Please explain, or fix the title / description.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/182743

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183064

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183438

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182600
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=abcde8950fee3e638a934eca3e79e25f5ba61d08
Submitter: Jenkins
Branch: master

commit abcde8950fee3e638a934eca3e79e25f5ba61d08
Author: Maciej Kwiek <email address hidden>
Date: Wed May 13 11:36:11 2015 +0200

    Add url_access_checker to network checker

    This change introduces new entry point for network_checker, which allows
    to check if it is possible to access urls give to the command.

    Usage: urlchecker check <url1> <url2> <url3> ...

    It will be used to check repository connectivity from slave nodes.

    python-requests is added to nailgun-net-check package requirements in
    both rpm spec and debian `control` file.

    Change-Id: Idc04f74ad7364dee452e9151391654f828e2342d
    Partial-Bug: #1439686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182743
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=12ef23583a958a5fecd1f9c8b27801ae979126ac
Submitter: Jenkins
Branch: master

commit 12ef23583a958a5fecd1f9c8b27801ae979126ac
Author: Maciej Kwiek <email address hidden>
Date: Wed May 13 18:12:38 2015 +0200

    Use urlaccesscheck to try to retrieve repos

    Add check_url_retrieval action to net_probe.rb. It uses urlaccesscheck
    from network checker to retrieve urls passed as parameter.

    Add check_repositories method to dispatcher. It uses above action via
    orchestrator to test given urls.

    Partial-Bug: #1439686
    Depends-On: Idc04f74ad7364dee452e9151391654f828e2342d
    Change-Id: Ifb28bd0a53abf07e48d41683d10f90546889b0e4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/183765

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Kwiek (<email address hidden>) on branch: master
Review: https://review.openstack.org/172954

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Kwiek (<email address hidden>) on branch: master
Review: https://review.openstack.org/172953

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183765
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=fac8f1af6bd3eb46fe2a26689b0d85f358934f7a
Submitter: Jenkins
Branch: master

commit fac8f1af6bd3eb46fe2a26689b0d85f358934f7a
Author: Dmitry Shulyak <email address hidden>
Date: Sat May 16 08:41:54 2015 +0300

    Add EDITOR variable in urlaccesschecker

    This variable required by cmd2 library to work.
    Without - it fails on bootstrap with traceback:

    File "/usr/lib/python2.6/site-packages/cmd2.py", line 424, in Cmd
    if subprocess.Popen(['which', editor])

    Change-Id: I061f88b65d7bc7181752cd076da4067df2f84131
    Related-Bug: 1439686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183438
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=23659819aeaf3f2da680a286b9ac7faf5ceef70e
Submitter: Jenkins
Branch: master

commit 23659819aeaf3f2da680a286b9ac7faf5ceef70e
Author: Dmitry Shulyak <email address hidden>
Date: Fri May 15 11:53:56 2015 +0300

    Add url access verification that will setup network

    Added command that will perform setup and teardown of
    required networking configuration.

    Network configuration will perform next things:

    - set interface up if required
    - create vlan tagged interface and set it up
    - add required ipv4 settings for interface
    - add default route

    After verification teardown will be performed.
    Teardown is best effort based - e.g we should not fail whole command
    if we cant fully execute teardown.

    Change-Id: I910c15c2b39a917eb8428bb69271b5dde364b639
    Partial-Bug: 1439686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183064
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=ded783e4623ff186aaad08ec76c8db09852af2b0
Submitter: Jenkins
Branch: master

commit ded783e4623ff186aaad08ec76c8db09852af2b0
Author: Maciej Kwiek <email address hidden>
Date: Thu May 14 11:54:55 2015 +0200

    Add CheckRepositoryConnectionFromSlavesTask

    This task is being invoked as a subtask of VerifyNetwork task, it uses
    calls astute check_repositories method which is introduced in the patch
    this change depends on.

    Receiverd action is added, to retrieve repository check result from
    Astute

    Change-Id: If820c72f2024ab18f8c4cd4177d658c1fa41caf3
    Partial-Bug: #1439686
    Depends-On: Ifb28bd0a53abf07e48d41683d10f90546889b0e4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183575
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=24eda075fe1c1c754fe9abb51e924cce43a3c7b9
Submitter: Jenkins
Branch: master

commit 24eda075fe1c1c754fe9abb51e924cce43a3c7b9
Author: Dmitry Shulyak <email address hidden>
Date: Fri May 15 18:39:35 2015 +0300

    Add repo availability verification with network setup

    Repo availability with and without setup will return
    response in next format:

      {uid=> '3', status=> 1, out=>{failed_urls=>[...]}, err=>"Error message"}

    Partial-Bug: 1439686
    Change-Id: I3216b97df5e7b8974ac7155d06446c4419d293a1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183526
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f8ba6bd748a95f70ff728cbdfab377674470d81b
Submitter: Jenkins
Branch: master

commit f8ba6bd748a95f70ff728cbdfab377674470d81b
Author: Dmitry Shulyak <email address hidden>
Date: Fri May 15 16:35:12 2015 +0300

    Repo availability verification using public network

    Generate config for repo availability setup with public network.
    Config should contain:
    - not used ip address from public network
    - vlan tag in case network is tagged (0 will be used otherwise)
    - gateway of public network
    - list of repo urls
    - iface where public network should be configured

    Partial-Bug: 1439686
    Change-Id: Ia588263730b9085650b10cc1f05edefcb43077cc

[Dmitry Pyzhov (dpyzhov)]

Every patch is in master

[Dmitry Tyzhnenko (dtyzhnenko)]

Verified on 6.1-521

{
    'build_id': '2015-06-08_06-13-27',
    'build_number': '521',
    'release_versions': {'2014.2.2-6.1': {'VERSION': {
        'build_id': '2015-06-08_06-13-27',
        'build_number': '521',
        'api': '1.0',
        'fuel-library_sha': 'f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d',
        'nailgun_sha': '4340d55c19029394cd5610b0e0f56d6cb8cb661b',
        'feature_groups': ['mirantis'],
        'openstack_version': '2014.2.2-6.1',
        'production': 'docker',
        'python-fuelclient_sha': '4fc55db0265bbf39c369df398b9dc7d6469ba13b',
        'astute_sha': '7766818f079881e2dbeedb34e1f67e517ed7d479',
        'fuel-ostf_sha': '7c938648a246e0311d05e2372ff43ef1eb2e2761',
        'release': '6.1',
        'fuelmain_sha': 'bcc909ffc5dd5156ba54cae348b6a07c1b607b24',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': 'f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d',
    'nailgun_sha': '4340d55c19029394cd5610b0e0f56d6cb8cb661b',
    'feature_groups': ['mirantis'],
    'openstack_version': '2014.2.2-6.1',
    'production': 'docker',
    'python-fuelclient_sha': '4fc55db0265bbf39c369df398b9dc7d6469ba13b',
    'astute_sha': '7766818f079881e2dbeedb34e1f67e517ed7d479',
    'fuel-ostf_sha': '7c938648a246e0311d05e2372ff43ef1eb2e2761',
    'release': '6.1',
    'fuelmain_sha': 'bcc909ffc5dd5156ba54cae348b6a07c1b607b24',
    }