python-setuptools installation failed during ansible provisioning

[Vedamurthy Joshi]

Build 4.0 3073

Below is the log

[root@ansible-runner playbooks]# ansible-playbook -i inventory/my-inventory site.yml
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default).
This feature will be removed in a
future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: While constructing a mapping from /home/root/jenkins/workspace/kubernetes-sanity/contrail-
ansible/playbooks/roles/storage/ceph_compute/tasks/ceph_prep_vars.yml, line 41, column 5, found a duplicate dict key (msg). Using last defined value only.

 [WARNING]: While constructing a mapping from /home/root/jenkins/workspace/kubernetes-sanity/contrail-
ansible/playbooks/roles/storage/ceph_compute/tasks/ceph_tune.yml, line 3, column 5, found a duplicate dict key (msg). Using last defined value only.

PLAY [Common system settings on base hosts] ************************************

TASK [setup] *******************************************************************
ok: [10.204.217.198]
ok: [10.204.217.197]
ok: [10.204.217.194]

TASK [Set flag to decide if docker is required] ********************************
ok: [10.204.217.194]
ok: [10.204.217.197]
ok: [10.204.217.198]

TASK [common : make sure /var/run/utmp exist] **********************************
ok: [10.204.217.198]
ok: [10.204.217.197]
ok: [10.204.217.194]

TASK [common : set /etc/hosts entry for hostname] ******************************
changed: [10.204.217.197]
changed: [10.204.217.198]
changed: [10.204.217.194]

TASK [common : Disable selinux for redhat systems] *****************************
changed: [10.204.217.197]
changed: [10.204.217.194]
changed: [10.204.217.198]

TASK [common : Findout if current machine is a container or a machine] *********
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [10.204.217.198]
changed: [10.204.217.194]
changed: [10.204.217.197]

TASK [common : Setting a fact whether the code is running in a docker container or not] ***
ok: [10.204.217.194]
ok: [10.204.217.197]
ok: [10.204.217.198]

TASK [common : Setting fact to override ansible_service_mgr for docker. This is to wrongly detect service manager as systemd when ubuntu container run on top of centos] ***
ok: [10.204.217.194]
ok: [10.204.217.197]
ok: [10.204.217.198]
TASK [common : Setting default for controller_components] **********************
ok: [10.204.217.194]
ok: [10.204.217.197]
ok: [10.204.217.198]

TASK [docker : resolve platform specific vars] *********************************
ok: [10.204.217.194] => (item=/home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/docker/vars/RedHat.yml)
ok: [10.204.217.197] => (item=/home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/docker/vars/RedHat.yml)
ok: [10.204.217.198] => (item=/home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/docker/vars/RedHat.yml)

TASK [docker : Include docker_install_method specific code] ********************
included: /home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/docker/tasks/package.yml for 10.204.217.194, 10.204.217.197, 10.204.217.198

TASK [docker : Add contrail apt repository if required] ************************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : Allow unauthenticated packages for contrail packages to work] ***
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : Pin contrail apt repo "contrail" release to priority 999] *******
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : Run apt-get update] *********************************************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]
TASK [docker : install docker using package] ***********************************
changed: [10.204.217.197]
changed: [10.204.217.198]
changed: [10.204.217.194]

TASK [docker : Adding EnvironmentFile to docker systemd file] ******************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : Adding EnvironmentFile to docker systemd file] ******************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : Adding DOCKER OPTIONS for docker init scripts] ******************
changed: [10.204.217.197]
changed: [10.204.217.194]
changed: [10.204.217.198]

TASK [docker : Added OPTIONS to docker start in /usr/lib/systemd/system/docker.service] ***
skipping: [10.204.217.197]
skipping: [10.204.217.194]
skipping: [10.204.217.198]

TASK [docker : Added OPTIONS to docker start in /lib/systemd/system/docker.service] ***
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]
TASK [docker : Adding Docker Insecure registries in RedHat] ********************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [docker : create root user docker config directory] ***********************
changed: [10.204.217.197]
changed: [10.204.217.198]
changed: [10.204.217.194]

TASK [docker : configuring docker access for the root user...] *****************
skipping: [10.204.217.194] => (item={u'm': u'0600', u't': u'cert.pem', u'f': u'~/pki/10.204.217.194.pem'})
skipping: [10.204.217.197] => (item={u'm': u'0600', u't': u'cert.pem', u'f': u'~/pki/10.204.217.197.pem'})
skipping: [10.204.217.194] => (item={u'm': u'0600', u't': u'ca.pem', u'f': u'~/pki/ca.pem'})
skipping: [10.204.217.197] => (item={u'm': u'0600', u't': u'ca.pem', u'f': u'~/pki/ca.pem'})
skipping: [10.204.217.198] => (item={u'm': u'0600', u't': u'ca.pem', u'f': u'~/pki/ca.pem'})
skipping: [10.204.217.198] => (item={u'm': u'0600', u't': u'cert.pem', u'f': u'~/pki/10.204.217.198.pem'})
skipping: [10.204.217.194] => (item={u'm': u'0400', u't': u'key.pem', u'f': u'~/pki/10.204.217.194-key.pem'})
skipping: [10.204.217.197] => (item={u'm': u'0400', u't': u'key.pem', u'f': u'~/pki/10.204.217.197-key.pem'})
skipping: [10.204.217.198] => (item={u'm': u'0400', u't': u'key.pem', u'f': u'~/pki/10.204.217.198-key.pem'})

TASK [docker : write /etc/profile.d/docker.sh] *********************************
changed: [10.204.217.194] => (item=etc/profile.d/docker.sh)
changed: [10.204.217.198] => (item=etc/profile.d/docker.sh)
changed: [10.204.217.197] => (item=etc/profile.d/docker.sh)

TASK [docker : launch docker...] ***********************************************
changed: [10.204.217.197]
changed: [10.204.217.194]
changed: [10.204.217.198]
RUNNING HANDLER [docker : restart docker] **************************************
changed: [10.204.217.198]
changed: [10.204.217.197]
changed: [10.204.217.194]

TASK [common : make sure /var/run/utmp exist] **********************************
ok: [10.204.217.194]
ok: [10.204.217.197]
ok: [10.204.217.198]

TASK [common : set /etc/hosts entry for hostname] ******************************
ok: [10.204.217.197]
ok: [10.204.217.194]
ok: [10.204.217.198]

TASK [common : Disable selinux for redhat systems] *****************************
changed: [10.204.217.194]
changed: [10.204.217.198]
changed: [10.204.217.197]

TASK [common : Findout if current machine is a container or a machine] *********
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..
This feature will be removed in version 2.4. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
skipping: [10.204.217.198]
skipping: [10.204.217.194]
skipping: [10.204.217.197]

TASK [common : Setting a fact whether the code is running in a docker container or not] ***
ok: [10.204.217.194]
ok: [10.204.217.198]
ok: [10.204.217.197]

TASK [common : Setting fact to override ansible_service_mgr for docker. This is to wrongly detect service manager as systemd when ubuntu container run on top of centos] ***
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [common : Setting default for controller_components] **********************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [node : fail in case of invalid os_release] *******************************
skipping: [10.204.217.194]
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [node : include OS specific code] *****************************************
skipping: [10.204.217.194]
included: /home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/node/tasks/host/RedHat.yml for 10.204.217.197, 10.204.217.198

TASK [node : Add contrail yum repository] **************************************
changed: [10.204.217.197]
changed: [10.204.217.198]

TASK [node : Disable firewalld on redhat systems] ******************************
changed: [10.204.217.198]
changed: [10.204.217.197]

TASK [node : Disable iptables on redhat systems] *******************************
fatal: [10.204.217.198]: FAILED! => {"changed": false, "failed": true, "msg": "systemd could not find the requested service \"'iptables'\": "}
...ignoring
fatal: [10.204.217.197]: FAILED! => {"changed": false, "failed": true, "msg": "systemd could not find the requested service \"'iptables'\": "}
...ignoring

TASK [node : Set correct hostname on centos - this should always be only hostname and not hostname.domainname format] ***
skipping: [10.204.217.197]
skipping: [10.204.217.198]

TASK [node : Create contrailctl config directory] ******************************
skipping: [10.204.217.194]
changed: [10.204.217.198]
changed: [10.204.217.197]

TASK [node : set /etc/hosts entry for all nodes with their hostname so that they can be reached with hostnames] ***
skipping: [10.204.217.194] => (item=10.204.217.194)
skipping: [10.204.217.194] => (item=10.204.217.197)
skipping: [10.204.217.194] => (item=10.204.217.198)
changed: [10.204.217.198] => (item=10.204.217.194)
changed: [10.204.217.197] => (item=10.204.217.194)
changed: [10.204.217.198] => (item=10.204.217.197)
changed: [10.204.217.197] => (item=10.204.217.197)
changed: [10.204.217.197] => (item=10.204.217.198)
changed: [10.204.217.198] => (item=10.204.217.198)

TASK [node : include] **********************************************************
included: /home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/node/tasks/pip.yml for 10.204.217.194, 10.204.217.197, 10.204.217.198
TASK [node : Install python-setuptools] ****************************************
changed: [10.204.217.194]
fatal: [10.204.217.197]: FAILED! => {"changed": false, "failed": true, "msg": "Failure talking to yum: failure: repodata/repomd.xml from contrail: [Errno 256] No more mirrors to try.\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\""}
fatal: [10.204.217.198]: FAILED! => {"changed": false, "failed": true, "msg": "Failure talking to yum: failure: repodata/repomd.xml from contrail: [Errno 256] No more mirrors to try.\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\"\nhttp://10.204.217.194:1567/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed connect to 10.204.217.194:1567; No route to host\""}

TASK [node : Make sure pip is installed] ***************************************
changed: [10.204.217.194]

TASK [node : Make sure docker-py is installed] *********************************
changed: [10.204.217.194]

TASK [node : include] **********************************************************
included: /home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/roles/node/tasks/repo/RedHat.yml for 10.204.217.194

TASK [node : Copy contrail yum repo image] *************************************
skipping: [10.204.217.194]

TASK [node : Load contrail yum repo image] *************************************
skipping: [10.204.217.194]

TASK [node : Launch yum repo container in RedHat] ******************************
skipping: [10.204.217.194]

TASK [node : Launch yum repo container in CentOS] ******************************
fatal: [10.204.217.194]: FAILED! => {"changed": false, "failed": true, "msg": "Error pulling 10.204.217.152:5000/contrail-repo-centos7 - code: None message: manifest unknown: manifest unknown"}
        to retry, use: --limit @/home/root/jenkins/workspace/kubernetes-sanity/contrail-ansible/playbooks/site.retry

PLAY RECAP *********************************************************************
10.204.217.194 : ok=26 changed=13 unreachable=0 failed=1
10.204.217.197 : ok=28 changed=14 unreachable=0 failed=1
10.204.217.198 : ok=28 changed=14 unreachable=0 failed=1

[root@ansible-runner playbooks]#

[Vedamurthy Joshi (vedujoshi)]

You could ignore this error in this case since python-setuptools install error was seen even after adding contrail-repo-centos7 in the docker registry
"
fatal: [10.204.217.194]: FAILED! => {"changed": false, "failed": true, "msg": "Error pulling 10.204.217.152:5000/contrail-repo-centos7 - code: None message: manifest unknown: manifest unknown"}"

[Ramprakash R (ramprakash)]

Can you please add the inventory details also so I could reproduce this issue?

[Vedamurthy Joshi (vedujoshi)]

Removed blocker since we have a workaround

For centos compute nodes , It is rm -f /etc/yum.repos.d/contrail* ; yum clean all and rerun ansible provisioning

[Ramprakash R (ramprakash)]

Not being observed on recent builds. Will keep this on watch mode and re-open if we see this again.