[centos64_build_1804] Broken Contrail Command install

[Cantemir Olaru]

DESCRIPTION
-------------
I'm following the official install guide for Contrail Command using a CentOS 7.5 ISO image (build 1804) and the install fails at step 4 while trying to download some Python dependencies from this site that uses TLS1.2 -> https://pypi.python.org/simple/docker-py/

Unfortunately CentOS 7 has deprecated the use of legacy SSL/TLS crypto hence the install fails because it cannot check the site's certificate.

The OpenSSL version used by CentOS7.5 is 1.0.2k and CentOS doesn't provide RPMs with newer OpenSSL versions. The only fix I could find is to download the latest OpenSSL source (1.1.1 as of writing this), then recompile it and link it instead of the old 1.0.2k version.

REPRO STEPS
-------------

1. Download a minimal CentOS7.5 ISO and install it on your VM/BMS.
I have used:

ISO image: CentOS-7-x86_64-Minimal-1804.iso
MD5: fabdc67ff3a1674a489953effa285dfd
VM: VMware ESXi / 2.3Ghz quad-core / 16GB RAM / 50GB HDD

2. Follow the install steps in the official Contrail Command guide to the letter:
https://www.juniper.net/documentation/en_US/contrail5.0/topics/example/install-contrail-command.html

3. When reaching step #4 in that guide you will hit the problem while the Ansible playbook tries to pull some dependencies from pypi.python.org:

[root@contrail-command ~]# docker run -t --net host -v /var/lib/docker/overlay2/48e3793362a49b179107996c74c09b6dcdb5942f5ee2d7f4602126f5371524db/diff/contrail-command-deployer/config/command_servers.yml:/command_servers.yml -d --privileged --name contrail_command_deployer hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214

[root@contrail-command ~]# docker logs -f contrail_command_deployer

[...]

TASK [install_packages : install docker-py] ************************************
fatal: [172.27.0.2]: FAILED! => {"changed": false, "cmd": "/usr/bin/pip2 install docker-py==1.10.6", "msg": "stdout: Collecting docker-py==1.10.6\n Could not fetch URL https://pypi.python.org/simple/docker-py/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) - skipping\n\n:stderr: Could not find a version that satisfies the requirement docker-py==1.10.6 (from versions: )\nNo matching distribution found for docker-py==1.10.6\n"}
       to retry, use: --limit @/var/tmp/contrail-commander/deploy.retry

PLAY RECAP *********************************************************************
172.27.0.2 : ok=13 changed=6 unreachable=0 failed=1
localhost : ok=4 changed=2 unreachable=0 failed=0

WORKAROUND - REINSTALL OPENSSL FROM SOURCE
---------------------------------------

[root@contrail-command ~]# openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017

Before you proceed you must make sure the time is synchronized via NTP.

[root@contrail-command ~]# yum -y update

[root@contrail-command ~]# yum -y install ntp ntpdate libtool perl-core zlib-devel wget

[root@contrail-command ~]# ntpdate -u us.pool.ntp.org

Download the latest OpenSSL, unpack it and install from source - https://www.openssl.org/source/

[root@contrail-command ~]# wget --no-check-certificate https://www.openssl.org/source/openssl-1.1.1.tar.gz
[root@contrail-command ~]# tar xvzf openssl-1.1.1.tar.gz
[root@contrail-command ~]# cd openssl-1.1.1
[root@contrail-command ~]# ./config --prefix=/usr/local/openssl --openssldir=/usr/local/openssl shared zlib
[root@contrail-command ~]# make
[root@contrail-command ~]# make install

Now statically link the new OpenSSL version(1.1.1) instead of the official CentOS one (1.0.2 at the time of writing this).
Create the following file:

[root@contrail-command ~]# vi /etc/profile.d/openssl.sh
And paste in the following content in that file:

pathmunge /usr/local/openssl/bin

Next create this ldconfig file to define the new path:

[root@contrail-command ~]# vi /etc/ld.so.conf.d/openssl-1.1.1.conf

And paste in the following content in that file:

/usr/local/openssl/lib

Relink all libraries:

[root@contrail-command ~]# ldconfig -v

Now confirm that OpenSSL is using the new version:

[root@contrail-command ~]# openssl version
OpenSSL 1.1.1 11 Sep 2018

Now back to installing Contrail Command, make sure that docker is running delete all previous imcomplete containers and rebuild them:

[root@contrail-command ~]# systemctl start docker

[root@contrail-command ~]# docker ps -a # get all container IDs

[root@contrail-command ~]# docker stop <container_id> # stop all containers

[root@contrail-command ~]# docker rm <container_id> # delete all containers

[root@contrail-command ~]# docker run -t --net host -v /var/lib/docker/overlay2/48e3793362a49b179107996c74c09b6dcdb5942f5ee2d7f4602126f5371524db/diff/contrail-command-deployer/config/command_servers.yml:/command_servers.yml -d --privileged --name contrail_command_deployer hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214

Follow the progress by monitoring the Docker logs:
[root@contrail-command ~]# docker logs -f contrail_command_deployer

DONE. Move on to Step 6 from the install guide.

[Leela Bharath Kumar Kassetti (klbharat)]

Are you seeing this error "There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED]" if you try to manually install any pip package or is this specific to docker-py installation.

P.S:
We are no longer installing docker-py as part of contrail-command installation.

[Leela Bharath Kumar Kassetti (klbharat)]

FYI, we use the same image "CentOS-7-x86_64-Minimal-1804.iso" and we never saw this issue. The OpenSSL version is 1.0.2 and we don't see any failures when installing docker-py.

It can be a setup related issue. We need setup details for further debugging.

[Leela Bharath Kumar Kassetti (klbharat)]

From Cantemir:

"I can’t share the testbed right now because it’s no longer in the broken state.
However, I did try to repro this a few times today and think I’ve found the issue:

No issue seen with these steps (as per our official procedure):
# yum -y install ansible-2.4.2.0 ntp yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum install -y docker-ce
# systemctl start docker
# docker login hub.juniper.net --username <user> --password <pass>
# docker pull hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214
(create /root/command_servers.yml)
# docker run -t --net host -v /root/command_servers.yml:/command_servers.yml -d --privileged --name contrail_command_deployer hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214

Issue seen with these when running the container:
# yum -y update
# yum -y install ansible-2.4.2.0 ntp yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum install -y docker-ce
# systemctl start docker
# docker login hub.juniper.net --username <user> --password <pass>
# docker pull hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214
(create /root/command_servers.yml)
# docker run -t --net host -v /root/command_servers.yml:/command_servers.yml -d --privileged --name contrail_command_deployer hub.juniper.net/contrail/contrail-command-deployer:5.0.1-0.214

I think I’m the only one that saw this because I always do a ‘yum -y update’ before installing anything on CentOS (trying to avoid obsolete packages and missing dependencies).

If you do a yum update at the very beginning you will likely see the same certificate issue. If you don’t you won’t.
Our install guide just asks to use CentOS7.5 but doesn’t warn against doing an update before the Command build… "

[Leela Bharath Kumar Kassetti (klbharat)]

To conclude:

The official documented procedure is working without any issues.

Issue is observed only when "yum -y update" command is given which installs latest packages. This step is not present in the official installation procedure.