Fuel 9 - Network Verification failed

[Tejas]

---------Environment information------
1 Fuel 9 Hyper-v VM.
1 Controller node
3 Compute Node
Neutron with VXLAN

Internet is connected to Fuel 9 server

--------------Expected result of network validation-------------

  Successfully validated network

--------------Actual result---------------------

Network validation successfully passes if we test only with compute nodes.

If we convert the compute node role to controller then i get following error

Verification failed.
Repo availability verification using public network failed on following nodes Controller (0a:ab).
Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/.
Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.

----------------Suspected Bug------------------------

 We have noticed that controller node is able to connect to internet. but not able to fetch public repo.

[Dmitry Klenov (dklenov)]

@Tejas, can you please provide diagnostic snapshot?

[Tejas (tsheth-p)]

I have created local mirror. but I will try to simulate the same and send the snapshot

[Tejas (tsheth-p)]

Let me know if following snapshot gives enough information.

https://s3.amazonaws.com/solutionfrontierbusinesssystems/fuel-snapshot-2016-09-17_05-56-50.tar.gz

  We have also noticed that deployment of MOS fails with local repos as well.

[Joey Officer (jofficer)]

I believe I'm running into a similar (if not same) issue.

New deployment on VMWare workstation
network setup as defined in Mirantis virtual box installation pre-reqs
create new environment
nodes are discovered
verify network - fails

Verification failed.

Repo availability verification using public network failed on following nodes controller main.

Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/.

Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.

error as reported in

2016-10-20 21:40:18.478 INFO [7f03c6b52740] (receiver) RPC method check_repositories_with_setup received: {"status": "ready", "progress": 100, "task_uuid": "b0ea765e-4936-4fb6-84ca-64292142a29a", "nodes": [{"status": 1, "uid": "1", "err": "Unexpected failure.\nTraceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/network.py\", line 218, in manage_network\n yield\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 76, in take_action\n CheckUrlsWithSetup, self).take_action(pa)\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 57, in take_action\n raise e\nUrlNotAvailable: {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/\"]}\n2016-10-20 21:40:18 ERROR (network) Unexpected failure.\nTraceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/network.py\", line 218, in manage_network\n yield\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 76, in take_action\n CheckUrlsWithSetup, self).take_action(pa)\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 57, in take_action\n raise e\nUrlNotAvailable: {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/\"]}\n{\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/\"]}\n2016-10-20 21:40:18 ERROR (app) {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/\"]}\n", "out": {"failed_urls": ["http://archive.ubuntu.com/ubuntu/", "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/"]}}]}

test on node-1 (of 4)

root@node-1:~# wget http://archive.ubuntu.com/ubuntu/
--2016-10-20 21:49:37-- http://archive.ubuntu.com/ubuntu/
Resolving archive.ubuntu.com (archive.ubuntu.com)... 91.189.88.152, 91.189.88.162, 91.189.88.149, ...
Connecting to archive.ubuntu.com (archive.ubuntu.com)|91.189.88.152|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1767 (1.7K) [text/html]
Saving to: ‘index.html.2’

100%[=======================================================================================================================================================================================>] 1,767 --.-K/s in 0s

2016-10-20 21:49:37 (224 MB/s) - ‘index.html.2’ saved [1767/1767]

[Bryan Taylor (btaylor-b)]

Same problem on real hardware using Fuel 10.

INFO [1888] Casting message to Nailgun:
{"method"=>"check_repositories_with_setup_resp",
 "args"=>
  {"task_uuid"=>"65d4ce36-febc-4600-957e-e30d41383819",
   "status"=>"ready",
   "progress"=>100,
   "nodes"=>
    [{:out=>
       {"failed_urls"=>
         ["http://archive.ubuntu.com/ubuntu/",
          "http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/"]},
      :err=>
       "Unexpected failure.\nTraceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/network.py\", line 218, in manage_network\n yield\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 76, in take_action\n CheckUrlsWithSetup, self).take_action(pa)\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 57, in take_action\n raise e\nUrlNotAvailable: {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/\"]}\n2016-12-12 22:11:40 ERROR (network) Unexpected failure.\nTraceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/network.py\", line 218, in manage_network\n yield\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 76, in take_action\n CheckUrlsWithSetup, self).take_action(pa)\n File \"/usr/lib/python2.7/dist-packages/url_access_checker/commands.py\", line 57, in take_action\n raise e\nUrlNotAvailable: {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/\"]}\n{\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/\"]}\n2016-12-12 22:11:40 ERROR (app) {\"failed_urls\": [\"http://archive.ubuntu.com/ubuntu/\", \"http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/\"]}\n",
      :status=>1,

[Vladimir Kuklin (vkuklin)]

Folks

This issue essentially means that the nodes cannot access the repositories, thus it means that they won't be able to download packages and the deployment will fail. Please check your networking configuration and test things by simply going to the bootstrapped nodes and running `urlaccesscheck` command to ensure your repositories are available. I will mark this bug as Invalid unless you provide sufficient info that repositories ARE really accessible from the nodes, but Fuel network verification shows you false-negative results.

[Bryan Taylor (btaylor-b)]

Really looks like a false negative:

[root@fuel ~]# urlaccesscheck check http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/
[returns prompt no error]

[root@fuel ~]# urlaccesscheck check http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/DFJDOIfjsdaf
{"failed_urls": ["http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/DFJDOIfjsdaf"]}
2017-01-05 21:01:15 ERROR (app) {"failed_urls": ["http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/DFJDOIfjsdaf"]}
{"failed_urls": ["http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0/DFJDOIfjsdaf"]}[root@fuel ~]#

[Gayathri Govindaraju (gaatri)]

Facing similar problem on hardware with fuel 9.0
[root@fuel ~]# fuel node list
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+---------------+-------------------+-------+---------------+--------+---------
 3 | discover | Untitled (c1:30) | 3 | 192.168.10.4 | 0c:c4:7a:75:c1:30 | | compute | 1 | 3
 1 | discover | Untitled (c2:30) | 3 | 192.168.10.9 | 0c:c4:7a:75:c2:30 | | controller | 1 | 3
 2 | discover | Untitled (c5:60) | 3 | 192.168.10.10 | 0c:c4:7a:75:c5:60 | | controller | 1 | 3
[root@fuel ~]# ssh root@192.168.10.9
Warning: Permanently added '192.168.10.9' (ECDSA) to the list of known hosts.
Last login: Thu Mar 30 21:23:29 2017
root@bootstrap:~#
root@bootstrap:~#
root@bootstrap:~#
root@bootstrap:~#
root@bootstrap:~# urlaccesscheck check http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/

[root@fuel ~]# ssh root@192.168.10.10
Warning: Permanently added '192.168.10.10' (ECDSA) to the list of known hosts.
Last login: Thu Mar 30 21:32:07 2017
root@bootstrap:~# urlaccesscheck check http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/
root@bootstrap:~#

Compute is fine, happens only in contoller.
even if I assign the node that works as a compute to controller, and then validate the network, the same error is displayed at Fuel UI

#################################################################
"Verification failed.
Repo availability verification using public network failed on following nodes Untitled (c2:30), Untitled (c5:60).
Following repos are not available - http://archive.ubuntu.com/ubuntu/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/.
Check your public network settings and availability of the repositories from public network. Please examine nailgun and astute logs for additional details.
"
###########################################################

rebooted the nodes as well and tried again. Same issue

Iam able to ping to outside world from these controller nodes
root@bootstrap:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=15.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=14.9 ms

[Manjunathkk (udaal)]

Looks like this is still an issue with 9.2 also. I have no issues in reaching the world from controller node and also urlaccesscheck has no issues in getting to those repos

Not sure why this was closed as invalid.