Network verification has failed.Repo availability verification using public network failed

[Anastasia Palkina]

1. Install fuel-master via fresh vbox-script
2. Install ISO #521
3. Create a new cluster and add required nodes
4. Start network verification
Verification failed.
Repo availability verification using public network failed on following nodes Untitled (1a:3f), Untitled (54:96), Untitled (93:77).
Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/
. Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.

I tried many times to restart network verification. Result is the same during 3 hours.

Connection to world exists from slave-nodes:
root@bootstrap:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=2.76 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=2.75 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=2.55 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=55 time=3.75 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 2.558/2.959/3.758/0.473 ms
root@bootstrap:~#

root@bootstrap:~# wget -v http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/
--2016-02-05 13:34:16-- http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/
Resolving mirror.fuel-infra.org (mirror.fuel-infra.org)... 5.43.231.47, 208.78.244.194
Connecting to mirror.fuel-infra.org (mirror.fuel-infra.org)|5.43.231.47|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html.2'

    [ <=> ] 537 --.-K/s in 0s

2016-02-05 13:34:16 (31.1 MB/s) - 'index.html.2' saved [537]

root@bootstrap:~#

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "521"
  build_id: "521"
  fuel-nailgun_sha: "bae6d0062e0825d81409a04bcb4979302f8c65ea"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "29829b131ca802830bc5a9a131c83cd0f43f702b"
  fuel-ostf_sha: "7bcddf18020f2d94a553a441ff57dff9632865df"
  fuel-mirror_sha: "c25d8931e30322ecf43246c8a957e376259b685c"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[Matthew Mosesohn (raytrac3r)]

Initial observations show networking is just fine, but Maksim is looking more deeply. It could be an issue in netcheck itself in python team.

[Maksim Malchuk (mmalchuk)]

didn't reproduced on MacOSX, VirtualBox 5.0.10r104061

[Dmitry Bilunov (dbilunov)]

For some reason eth1 was down on master node. It caused network verification to fail on controllers:

GET http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/
Current routing table is:
default via 172.16.0.1 dev enp0s8
10.20.0.0/24 dev enp0s3 proto kernel scope link src 10.20.0.6
172.16.0.0/24 dev enp0s8 proto kernel scope link src 172.16.0.2
Unexpected failure: <class 'requests.exceptions.ConnectionError'>.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/url_access_checker/network.py", line 217, in manage_network
    yield
  File "/usr/lib/python2.7/dist-packages/url_access_checker/commands.py", line 63, in take_action
    CheckUrlsWithSetup, self).take_action(pa)
  File "/usr/lib/python2.7/dist-packages/url_access_checker/commands.py", line 41, in take_action
    api.check_urls(parsed_args.urls, timeout=parsed_args.timeout)
  File "/usr/lib/python2.7/dist-packages/url_access_checker/api.py", line 41, in check_urls
    u, proxies=proxies, timeout=timeout), urls)
  File "/usr/lib/python2.7/dist-packages/url_access_checker/api.py", line 41, in <lambda>
    u, proxies=proxies, timeout=timeout), urls)
  File "/usr/lib/python2.7/dist-packages/url_access_checker/api.py", line 68, in _get_response_tuple
    return _get_http_response_tuple(url, proxies, timeout)
  File "/usr/lib/python2.7/dist-packages/url_access_checker/api.py", line 85, in _get_http_response_tuple
    response = requests.get(url) # , proxies=proxies, timeout=timeout * 1000)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='mirror.fuel-infra.org', port=80): Max retries exceeded with url: /mos-repos/ubuntu/8.0/ (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host')))

Network verification test restores a normal routing table upon completion, so interactively running wget won't reproduce the problem.

[Dmitry Bilunov (dbilunov)]

I cannot reproduce this bug on Linux 4.4.0 and VirtualBox 5.0.14. All the interfaces of the master node (eth0..2) are up and running.

[Dmitry Pyzhov (dpyzhov)]

Moving medium bug to 9.0 because we are in HCF for 8.0

[Maksim Malchuk (mmalchuk)]

This is floating issue caused by this removal: https://github.com/openstack/fuel-main/commit/27634ecd26834d5242b7707b9e463ea90b9cf1bf#diff-dced5bab835c6988664d60f8dfe2c6e0L252

[Maksim Malchuk (mmalchuk)]

Fix for this bug postponed because of https://review.openstack.org/#/q/topic:bp/get-rid-docker-containers
Will be fixed the right way after an ISO without docker containers will pass the tests.

[Maksim Malchuk (mmalchuk)]

We have an ISO without docker containers which pass the tests: https://product-ci.infra.mirantis.net/view/9.0-liberty/job/9.0-liberty.all/196/
This bug is marked as Hight because we need to fix this urgently for new ISOs.

[Serhii Ovsianikov (sovsianikov)]

Reproduced on Linux Ubuntu 14.04.2 LTS, with 8.0 RC1 ISO #529

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/278548

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278548
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=fad5dbb041b1188b4f55cbca2c78b86967b078ec
Submitter: Jenkins
Branch: master

commit fad5dbb041b1188b4f55cbca2c78b86967b078ec
Author: Maksim Malchuk <email address hidden>
Date: Wed Feb 10 21:00:03 2016 +0300

    Increase timeouts for the Cobbler container restart

    This commit increase timeouts for 'expect' to wait for the
    command prompt after the Cobbler container restart which
    in some cases takes too long time.

    Change-Id: Ic9861cc08f6411f8b126a02432422ad07f49d22a
    Closes-Bug: #1542320

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/278883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278883
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=d605bcbabf315382d56d0ce8143458be67c53434
Submitter: Jenkins
Branch: stable/8.0

commit d605bcbabf315382d56d0ce8143458be67c53434
Author: Maksim Malchuk <email address hidden>
Date: Wed Feb 10 21:00:03 2016 +0300

    Increase timeouts for the Cobbler container restart

    This commit increase timeouts for 'expect' to wait for the
    command prompt after the Cobbler container restart which
    in some cases takes too long time.

    Change-Id: Ic9861cc08f6411f8b126a02432422ad07f49d22a
    Closes-Bug: #1542320
    (cherry picked from commit fad5dbb041b1188b4f55cbca2c78b86967b078ec)

[Anastasia Palkina (apalkina)]

Verified on ISO #529 for 8.0

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[root@fuel ~]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 172.17.42.1 netmask 255.255.0.0 broadcast 0.0.0.0
        ether 02:42:21:a2:2d:c8 txqueuelen 0 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.20.0.2 netmask 255.255.255.0 broadcast 10.20.0.255
        inet6 fe80::a00:27ff:fe60:90e6 prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:60:90:e6 txqueuelen 1000 (Ethernet)
        RX packets 318240 bytes 45900152 (43.7 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 2364591 bytes 3537159401 (3.2 GiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 172.16.0.1 netmask 255.255.255.0 broadcast 172.16.0.255
        inet6 fe80::a00:27ff:fed0:7209 prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:d0:72:09 txqueuelen 1000 (Ethernet)
        RX packets 1041 bytes 339279 (331.3 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 478 bytes 32446 (31.6 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.200.15 netmask 255.255.255.0 broadcast 192.168.200.255
        inet6 fe80::a00:27ff:fe1b:b826 prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:1b:b8:26 txqueuelen 1000 (Ethernet)
        RX packets 596835 bytes 540777101 (515.7 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 195208 bytes 11988245 (11.4 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 0 (Local Loopback)
        RX packets 148694 bytes 286964193 (273.6 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 148694 bytes 286964193 (273.6 MiB)
        TX ...

[TatyanaGladysheva (tgladysheva)]

Verified on:
[root@fuel ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 482
cat /etc/fuel_build_number:
 482
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0

Actual result:
Network verification is successful.
[root@fuel ~]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
        ether 02:42:3b:81:3c:37 txqueuelen 0 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.20.0.2 netmask 255.255.255.0 broadcast 10.20.0.255
        inet6 fe80::a00:27ff:fe27:71dc prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:27:71:dc txqueuelen 1000 (Ethernet)
        RX packets 183017 bytes 24938845 (23.7 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 648981 bytes 961041283 (916.5 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 172.16.0.1 netmask 255.255.255.0 broadcast 172.16.0.255
        inet6 fe80::a00:27ff:fecf:1ddb prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:cf:1d:db txqueuelen 1000 (Ethernet)
        RX packets 12 bytes 1063 (1.0 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 70 bytes 5579 (5.4 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.200.15 netmask 255.255.255.0 broadcast 192.168.200.255
        inet6 fe80::a00:27ff:fecf:cfff prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:cf:cf:ff txqueuelen 1000 (Ethernet)
        RX packets 203204 bytes 277835563 (264.9 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 27848 bytes 1819512 (1.7 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 0 (Local Loopback)
        RX packets 36052 bytes 26086806 (24.8 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 36052 bytes 26086806 (24.8 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[root@fuel ~]#

[Tejas (tsheth-p)]

I am also facing the same issue but with physical deployment.

Controller is able to connect to internet but network validation fails to check the repo.

Verification failed.
Repo availability verification using public network failed on following nodes Controller (0a:ab).
Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/.
Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.

[Maksim Malchuk (mmalchuk)]

Tejas, this is another issue, feel free to open new bug and provide diagnostic snapshot for investigation.