SM provision hangs on keystone API failure at openstack node

[wenqing liang]

4.0.1.0-89~mitaka.

            "provision_role_sequence": "{'completed': [(u'server2', 'keepalived', '2017_09_26__19_50_35'), (u'server1', 'keepalived', '2017_09_26__19_50_36'), (u'server3', 'keepalived', '2017_09_26__19_55_03'), (u'server2', 'haproxy', '2017_09_26__19_55_19'), (u'server1', 'haproxy', '2017_09_26__19_55_20'), (u'server3', 'haproxy', '2017_09_26__19_55_20'), (u'server1', 'openstack', '2017_09_26__20_02_26'), (u'server3', 'openstack', '2017_09_26__20_24_15')], 'steps': [[(u'server2', u'openstack')], [(u'server2', u'pre_exec_vnc_galera')], [(u'server1', u'pre_exec_vnc_galera')], [(u'server3', u'pre_exec_vnc_galera')], [(u'server2', u'post_exec_vnc_galera')], [(u'server1', u'post_exec_vnc_galera')], [(u'server3', u'post_exec_vnc_galera')], [(u'server1', 'post_provision'), (u'server2', 'post_provision'), (u'server3', 'post_provision')]]}",

Sep 26 22:22:59 server2 puppet-agent[5203]: Not managing Keystone_role[_member_] due to earlier Keystone API failures.
Sep 26 22:22:59 server2 puppet-agent[5203]: (/Stage[openstack]/Contrail::Profile::Openstack::Provision/Keystone_role[_member_]/ensure) change from absent to present failed: Not managing Keystone_role[_member_] due to earlier Keystone API failures.

Sep 26 20:03:32 server2 puppet-agent[6685]: (/Stage[openstack]/Contrail::Profile::Openstack::Provision/Keystone_user[admin::Default]) Could not evaluate: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Certificate did not match expected hostname: 10.10.0.100. Certificate: {'notAfter': 'Sep 14 19:47:00 2027 GMT', 'subjectAltName': (('DNS', 'server2'), ('DNS', 'server2.englab.juniper.net'), ('IP Address', '10.10.0.5'), ('IP Address', '10.0.0.5')), 'subject': ((('commonName', u'server2'),),)}

[wenqing liang (wliang)]

Certificates generated for servers doesn’t have ip-address of internal-vip and causing the failure. This is blocking r4.0 mitaka.

[Jeba Paulaiyan (jebap)]

This is with Keystone v3

[Dheeraj Gautam (dgautam)]

ISSUE:
With keystone SSL + HA, nodes not able to connect keystone as services are connecting to keystone saying certificate can’t be trusted. Other cases, there won’t be any issue.

ROOT CAUSE:
Certificates generated have mgmt-ip and ctrl-data ip-addresses of the hosts. But services connect to internal/external vip. SSL handshake fails as certificates can’t be validated by clients as certificates doesn’t have external/internal vip.

Fix:
1. add VIPs to all openssl.conf of all openstack nodes  during certificate generation
or
Use –insecure option inside of openstack puppet code (not recommended)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36484
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36485
Submitter: Dheeraj Gautam (<email address hidden>)

[Abhay Joshi (abhayj)]

From: Ignatious Johnson <email address hidden>
Date: Thursday, August 10, 2017 at 10:59 AM
To: "<email address hidden>" <email address hidden>, Vinay Vithal Mahuli <email address hidden>
Cc: Nagendra Maynattamai <email address hidden>, Karl Klashinsky <email address hidden>, Sachin Bansal <email address hidden>, Abhay Joshi <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Discrepancy in cache/mirror ( trusty-mitaka-updates)

Correcting Vinay’s Email ID.

From: Ignatious Johnson <email address hidden>
Date: Thursday, August 3, 2017 at 11:53 AM
To: Dheeraj Gautam <email address hidden>, Vinay Mahuli <email address hidden>
Cc: Nagendra Prasath Maynattamai Prem Chandran <email address hidden>, Karl Klashinsky <email address hidden>, Sachin Bansal <email address hidden>, Abhay Joshi <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Discrepancy in cache/mirror ( trusty-mitaka-updates)

Hi Vinay,

Any update on this?

Thanks,
Ignatious

From: Dheeraj Gautam <email address hidden>
Date: Tuesday, July 25, 2017 at 5:12 PM
To: Ignatious Johnson <email address hidden>, Vinay Mahuli <email address hidden>
Cc: Nagendra Prasath Maynattamai Prem Chandran <email address hidden>, Karl Klashinsky <email address hidden>, Sachin Bansal <email address hidden>, Abhay Joshi <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Discrepancy in cache/mirror ( trusty-mitaka-updates)

+Jeba

This is impacting SSL functionality of containers/openstack.

-Dheeraj

From: "<email address hidden>" <email address hidden>
Date: Monday, July 24, 2017 at 5:06 PM
To: Ignatious Johnson <email address hidden>, Vinay Mahuli <email address hidden>
Cc: Nagendra Maynattamai <email address hidden>, Karl Klashinsky <email address hidden>, Sachin Bansal <email address hidden>, Abhay Joshi <email address hidden>
Subject: Re: Discrepancy in cache/mirror ( trusty-mitaka-updates)

Has this issue been addressed?

root@server2:~# dpkg -l | grep urllib3
ii python-urllib3 1.13.1-2~cloud0 all HTTP library with thread-safe connection pooling for Python
ii python-urllib3-whl 1.13.1-1~cloud0 all HTTP library with thread-safe connection pooling

-Dheeraj

From: Ignatious Johnson <email address hidden>
Date: Thursday, June 29, 2017 at 5:01 PM
To: Vinay Mahuli <email address hidden>
Cc: Nagendra Maynattamai <email address hidden>, Karl Klashinsky <email address hidden>, Sachin Bansal <email address hidden>, Abhay Joshi <email address hidden>, "<email address hidden>" <email address hidden>
Subject: Discrepancy in cache/mirror ( trusty-mitaka-updates)

Hi Vinay,

I see that the upstream mitaka cache has different version(Version: 1.13.1-2ubuntu0.16.04.1~cloud0) of urllib3 and we use older version(Version: 1.13.1-2~cloud0) in R4.0.
However in R3.1.1.x we have the correct latest version.

Upstream: http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/trusty-updates/mitaka/main/binary-amd64/Packages
R4.0: https://github.com/Juniper/contrail-packaging/blob/R4.0/build/package_configs/ubuntu1404/mitaka/depends_contrail_openstac...

[Abhay Joshi (abhayj)]

From: Dheeraj Gautam <email address hidden>
Date: Thursday, October 12, 2017 at 3:47 PM
To: Abhay Joshi <email address hidden>
Subject: Status of Keystone SSL + HA bug : https://bugs.launchpad.net/juniperopenstack/+bug/1719752

Status of Keystone SSL Issue:

With my recent changes, I am able to provision Keystone SSL + Openstack HA. But I got stuck at contrail-api provisioning as contrail-api was not able to connect to keystone and reporting following errors:

10/12/2017 11:21:55 AM [contrail-api]: Connect error to contrail api 1500 tries: Unable to connect to keystone for authentication. Exception hostname '5.1.1.100' doesn't match either of 'a5d02e40', 'a5d02e40.englab.juniper.net'
10/12/2017 11:23:38 AM [contrail-api]: Connect error to contrail api 1600 tries: Unable to connect to keystone for authentication. Exception hostname '5.1.1.100' doesn't match either of 'a5d02e40', 'a5d02e40.englab.juniper.net'

This is due to the old libraries on the contrail-controller container. Contrail-api is not able to validate the certificate based on SNI extension and fails to connect to keystone.

-Dheeraj

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36491
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36496
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36497
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36485
Committed: http://github.com/Juniper/contrail-puppet/commit/91260d8fe206d7e47238515bfae008db56e4956d
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 91260d8fe206d7e47238515bfae008db56e4956d
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:50:07 2017 -0700

change haproxy mode to tcp and export cafile for openstackrc_v3 file

Change-Id: Iafe6c38640fdc593c5cc77ca0dba519541f439c1
Partial-Bug: #1719752

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36484
Committed: http://github.com/Juniper/contrail-server-manager/commit/e901997f9d5299f6a262286bcab9236e42b9f9d3
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit e901997f9d5299f6a262286bcab9236e42b9f9d3
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:42:28 2017 -0700

create certs with vip ip-addresses for HA case

Partial-Bug: #1719752

Changes made:
============
1. delete host certificates on reimaging server
2. pass cluster informaration during certificate generation
3. pull internal vip and external vip out of cluster configuration

Change-Id: I823ab000ccf1cbe64a1c44c2cfe7012f741cda66

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36857
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36491
Committed: http://github.com/Juniper/contrail-packaging/commit/64f414fcd96855bc9d29eea66b9b35e72413dc8d
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 64f414fcd96855bc9d29eea66b9b35e72413dc8d
Author: Vinay Mahuli <email address hidden>
Date: Thu Oct 12 21:26:05 2017 -0700

update python-urllib3 for keystone API failure

Change-Id: I2bdf2a3d4de72ac549661068bd219be69db8cb3d
Closes-Bug: 1719752

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36857
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36857
Committed: http://github.com/Juniper/contrail-packaging/commit/a283250709bfcac19f3bfc575ce05c3400989d68
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit a283250709bfcac19f3bfc575ce05c3400989d68
Author: Vinay Mahuli <email address hidden>
Date: Thu Oct 26 11:48:25 2017 +0530

Add new version of python-urllib3 for keystone API failure

Closes-Bug: 1719752

PATCH 2:
1. fix server-manager as well
2. change Depends one per line

Change-Id: I3cea6eca74243bdce9e5869e196638790d6f5f26

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36949
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36971
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36949
Committed: http://github.com/Juniper/contrail-packaging/commit/b1d3a08469edc0f4ee711b25054946db60f1d2b5
Submitter: Zuul (<email address hidden>)
Branch: master

commit b1d3a08469edc0f4ee711b25054946db60f1d2b5
Author: Vinay Mahuli <email address hidden>
Date: Thu Oct 26 11:48:25 2017 +0530

Add new version of python-urllib3 for keystone API failure

Closes-Bug: 1719752

PATCH 2:
1. fix server-manager as well
2. change Depends one per line

Change-Id: I3cea6eca74243bdce9e5869e196638790d6f5f26
(cherry picked from commit a283250709bfcac19f3bfc575ce05c3400989d68)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36971
Committed: http://github.com/Juniper/contrail-packaging/commit/cf2d839a02b9a9731d9be468f63f5392369fd95d
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit cf2d839a02b9a9731d9be468f63f5392369fd95d
Author: Vinay Mahuli <email address hidden>
Date: Thu Oct 26 11:48:25 2017 +0530

Add new version of python-urllib3 for keystone API failure

Closes-Bug: 1719752

PATCH 2:
1. fix server-manager as well
2. change Depends one per line

PATCH 3:
fixed the conflict.

(cherry picked from commit a283250709bfcac19f3bfc575ce05c3400989d68)

Conflicts:
 build/package_configs/ubuntu1404/mitaka/depends_contrail_openstack_packages.cfg

Change-Id: I3cea6eca74243bdce9e5869e196638790d6f5f26

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38655
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38656
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38655
Committed: http://github.com/Juniper/contrail-server-manager/commit/b2df63ea827520ada0f63583f78764d5b7247154
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit b2df63ea827520ada0f63583f78764d5b7247154
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:42:28 2017 -0700

create certs with vip ip-addresses for HA case

Partial-Bug: #1719752

Changes made:
============
1. delete host certificates on reimaging server
2. pass cluster informaration during certificate generation
3. pull internal vip and external vip out of cluster configuration

Change-Id: I823ab000ccf1cbe64a1c44c2cfe7012f741cda66
(cherry picked from commit e901997f9d5299f6a262286bcab9236e42b9f9d3)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38656
Committed: http://github.com/Juniper/contrail-server-manager/commit/bb93ee358f638730b2e13b7fc00539ad00ccc334
Submitter: Zuul (<email address hidden>)
Branch: master

commit bb93ee358f638730b2e13b7fc00539ad00ccc334
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:42:28 2017 -0700

create certs with vip ip-addresses for HA case

Partial-Bug: #1719752

Changes made:
============
1. delete host certificates on reimaging server
2. pass cluster informaration during certificate generation
3. pull internal vip and external vip out of cluster configuration

Change-Id: I823ab000ccf1cbe64a1c44c2cfe7012f741cda66
(cherry picked from commit e901997f9d5299f6a262286bcab9236e42b9f9d3)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38781
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38782
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38781
Committed: http://github.com/Juniper/contrail-puppet/commit/c26c74183f29a46079c5cf9e0ebef56131c6a320
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit c26c74183f29a46079c5cf9e0ebef56131c6a320
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:50:07 2017 -0700

change haproxy mode to tcp and export cafile for openstackrc_v3 file

Change-Id: Iafe6c38640fdc593c5cc77ca0dba519541f439c1
Partial-Bug: #1719752
(cherry picked from commit 91260d8fe206d7e47238515bfae008db56e4956d)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38782
Committed: http://github.com/Juniper/contrail-puppet/commit/9881627990868ebfe4b196a18d60988a49c70e20
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9881627990868ebfe4b196a18d60988a49c70e20
Author: Dheeraj Gautam <email address hidden>
Date: Thu Oct 12 14:50:07 2017 -0700

change haproxy mode to tcp and export cafile for openstackrc_v3 file

Change-Id: Iafe6c38640fdc593c5cc77ca0dba519541f439c1
Partial-Bug: #1719752
(cherry picked from commit 91260d8fe206d7e47238515bfae008db56e4956d)