Deployment cluster with Ubuntu bootstrap fail with HTTPConnectionPool error

[Artem Hrechanychenko]

Steps to reproduce:
1)Provisioning master node
2)Change bootstrap to Ubuntu in Fuel menu
3)bootstraping slaves
4)Deploy any cluster

Acture result:
Deployment cluster with Ubuntu bootstrap fail with HTTPConnectionPool error

 [7f3eb8ff0740] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 57, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1168, in execute
    failed_repositories = cls._get_failed_repositories(task)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1184, in _get_failed_repositories
    responses = cls._get_responses(urls)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1190, in _get_responses
    return map(requests.get, urls)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='archive.ubuntu.com', port=80): Max retries exceeded with url: /ubuntu/ (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

{"build_id": "2015-07-29_11-51-41", "build_number": "708", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-07-29_11-51-41", "build_number": "708", "api": "1.0", "fuel-library_sha": "76dc59fa08ea78eda51e8774797fae88ec525740", "nailgun_sha": "9f1afe861b2149446ce97595c01ef38478bf2d2a", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "1512b9af6b41cc95c4d891c593aeebe0faca5a63", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "355c08a04917f047b88f66242767049d2b1d0ff0", "production": "docker", "python-fuelclient_sha": "f04e6c46783ecd6000df31b61b6749da66d4d828", "astute_sha": "34e0493afa22999c4a07d3198ceb945116ab7932", "fuel-ostf_sha": "888baa3eab00242bf98eccdda36144deb7683bb3", "release": "7.0", "fuelmain_sha": "de5b333815f8541224c6726dc8446ffc7fb18b5b"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "76dc59fa08ea78eda51e8774797fae88ec525740", "nailgun_sha": "9f1afe861b2149446ce97595c01ef38478bf2d2a", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "1512b9af6b41cc95c4d891c593aeebe0faca5a63", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "355c08a04917f047b88f66242767049d2b1d0ff0", "production": "docker", "python-fuelclient_sha": "f04e6c46783ecd6000df31b61b6749da66d4d828", "astute_sha": "34e0493afa22999c4a07d3198ceb945116ab7932", "fuel-ostf_sha": "888baa3eab00242bf98eccdda36144deb7683bb3", "release": "7.0", "fuelmain_sha": "de5b333815f8541224c6726dc8446ffc7fb18b5b"}

[Artem Hrechanychenko (agrechanichenko)]

KVM environment

[Alexei Sheplyakov (asheplyakov)]

The error is raised by nailgun, specifically, by the code which checks if the (APT) repositories
are available from the master node [1]. It has nothing to do with bootstrap.

Most likely the problem is caused by wrong networking configuration. Please login to the master
node and run the following command:

dockerctl shell nailgun curl -O http://archive.ubuntu.com/ubuntu/dists/trusty/Release

and post the output here.

[1] https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/task/task.py#L1182-L1190

[Alexei Sheplyakov (asheplyakov)]

Also could you please attach dnsmask log?
To acquire this log run

 dockerctl shell cobbler cat /var/log/dnsmasq.log

[Artem Hrechanychenko (agrechanichenko)]

dnsmasq log

[Artem Hrechanychenko (agrechanichenko)]

[root@nailgun ~]# dockerctl shell nailgun curl -O http://archive.ubuntu.com/ubuntu/dists/trusty/Release
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 58512 100 58512 0 0 141k 0 --:--:-- --:--:-- --:--:-- 191k

[Alexei Sheplyakov (asheplyakov)]

Jul 30 10:56:09 dnsmasq[1366]: query[A] archive.ubuntu.com from 10.109.0.2
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.91.13
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.92.201
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.92.200
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.91.24
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.91.23
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.91.15
Jul 30 10:56:09 dnsmasq[1366]: cached archive.ubuntu.com is 91.189.91.14
Jul 30 10:56:09 dnsmasq[1366]: query[AAAA] archive.ubuntu.com from 10.109.0.2
Jul 30 10:56:09 dnsmasq[1366]: forwarded archive.ubuntu.com to 10.109.0.1
Jul 30 10:56:09 dnsmasq[1366]: reply archive.ubuntu.com is 2001:67c:1360:8c01::19
Jul 30 10:56:09 dnsmasq[1366]: reply archive.ubuntu.com is 2001:67c:1360:8c01::18

So it looks like DNS has been configured properly on the master node.
Yet nailgun complains about Ubuntu mirror being unavailable.

2015-07-30 10:56:09.519 DEBUG [7f3eb8ff0740] (mule) MULE STARTING for ApplyChangesTaskManager._execute_async
2015-07-30 10:56:09.664 INFO [7f3eb8ff0740] (manager) ApplyChangesTask: execute async starting for task 6
2015-07-30 10:56:11.911 ERROR [7f3eb8ff0740] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 57, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1168, in execute
    failed_repositories = cls._get_failed_repositories(task)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1184, in _get_failed_repositories
    responses = cls._get_responses(urls)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 1190, in _get_responses
    return map(requests.get, urls)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='archive.ubuntu.com', port=80): Max retries exceeded with url: /ubuntu/ (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

[Artem Hrechanychenko (agrechanichenko)]

looks like some network problem in Czech lab

[abushman (adam-bushman-g)]

OK I think I figure it out for some reason the /etc/fuel/asutue.yamal had 8.8.8.8 as the dns I changed it for a try to
192.168.1.1 (my local router gw ) then restarted fuel master and now the deploy is proceeding

Wish the error included some message like failure in dns resolution as called using /etc/fuel/astute.yaml.

Can you chain DNS in the below file like DNS_UPSTREAM with more dns ips?

"HOSTNAME": "fuel"
"DNS_DOMAIN": "domain.tld"
"DNS_SEARCH": "domain.tld"
"DNS_UPSTREAM": "192.168.1.1"
"NTP1": "0.fuel.pool.ntp.org"
"NTP2": "1.fuel.pool.ntp.org"
"NTP3": "2.fuel.pool.ntp.org"
"ADMIN_NETWORK":

[abushman (adam-bushman-g)]

Well also had to add a local ntp server of fuel master 10.20.0.2
but finally got a success!! seems like the verify network utility should do a bit more
cheers
adam