OpenContrail requires API certificate key on compute nodes

[Ante Karamatić]

Contrail uses python's requests module for handling HTTP(S) requests. Nature of this module is to trust only CAs that are located in Mozilla trust "store". This means that using in-house CAs requires usage of 'verify' parameter when calling requests.get or requests.post.

And, Contrail does this. But the problem is that Contrail expect CA, certificate and the key. This is defined by tree variables in vnc_api_lib.init:
 - certfile
 - keyfile
 - cafile

Documentation I found (https://github.com/Juniper/contrail-controller/wiki/SSL-configuration-for-API,-neutron-server-and-openstack-keystone-in-Contrail) is a bit vague on how to configure these three variables. Unless I'm reading this wrong, 'contrailbundle.pem' consists of API private key, API certificate and general CA and this bundle is used for API access. At the same time, same files (same key!) is used in vnc_api_lib.ini on compute nodes. This is at least undesired behaviour.

Looking at the code:

        url = "%s://%s:%s%s" % (self._authn_protocol, self._authn_server, self._authn_port,
                                  self._authn_url)
        new_headers = headers or {}
        try:
           if self._ksinsecure:
                response = requests.post(url, data=self._authn_body,
                                     headers=self._DEFAULT_AUTHN_HEADERS, verify=False)
           elif not self._ksinsecure and self._use_ks_certs:
                response = requests.post(url, data=self._authn_body,
                                         headers=self._DEFAULT_AUTHN_HEADERS, verify=self._kscertbundle)
           else:
                response = requests.post(url, data=self._authn_body,
                                         headers=self._DEFAULT_AUTHN_HEADERS)

If _use_ks_certs is not defined, communication falls back to whatever python requests considers secure. And, again, those are only certificates from Mozilla store - so no in-house CAs are possible. This can be rectifying by adjusting this part of vnc_api_lib.py:

            self._use_ks_certs=False
            if kscertfile and kskeyfile and kscafile \
               and self._authn_protocol == 'https':
                   certs=[kscertfile, kskeyfile, kscafile]
                   self._kscertbundle=utils.getCertKeyCaBundle(VncApi._DEFAULT_KS_CERT_BUNDLE,certs)
                   self._use_ks_certs=True

into something like:

            self._use_ks_certs=False
            if kscafile and self._authn_protocol == 'https':
                certs=[kscafile]
                if kskeyfile and kscertfile:
                    certs=[kscertfile, kskeyfile, kscafile]
                self._kscertbundle=utils.getCertKeyCaBundle(VncApi._DEFAULT_KS_CERT_BUNDLE,certs)
                self._use_ks_certs=True

This would allow defining only CA in vnc_api_lib.ini and would allow standard application of certificate authority. Same problem exists in contrail_plugin.py.

[Sanju Abraham (asanju)]

Could you please explain the use case of the application you are building. Is this application browser based and trying to connect with API server that has SSL enabled?

We have drawn a parallel to the openstack params and defined key, cert and ca and mandated key file in case the server (keystone) has cert_required and assumed that it should be passed always, even if we are providing the same file as the cert/ca in the keyfile parameter.

The cert_required param is deprecated in liberty and mitaka and hence we could also relax the keyfile parameter in the clients connecting to the API server, pretty much like what you have done above.

[Ante Karamatić (ivoks)]

I'm not building an app, just trying to use Contrail with OpenStack configured with SSL and company-wide CA.

With current code, if I'd like contrail to use SSL from compute node to keystone, I have two options. I can specify nothing in vnc_api_lib.ini or specify key, cert and CA. If I don't specify anything, then python will (or it should) read system wide CA's. If python is well installed and configured, then this will just work. This is not the case with Juniper Contrail OpenStack, which installs python-certifi with broken CA path (on Ubuntu, it overrides system's default and points to CA in /usr/share/python2.7/dist-packages... instead of pointing to /etc/ssl/certs/ca-certificates.crt).

So, now I'm on compute node that also uses vnc_api_lib.py and the only way to trigger requests.post(url, data=self._authn_body, headers=self._DEFAULT_AUTHN_HEADERS, verify=self._kscertbundle) is to specify a keyfile too (_use_ks_certs is set only when all three are defined).

keyfile config option, according to link above is a private key of keystone.

Therefore, instead of relying only on CA or on CA and the cert, clients need to specify keystone's private key. Again, this is not actually required, but contrail's vnc_api_lib for some reason requires it.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25066
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25068
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25069
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25070
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25071
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25072
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25069
Committed: http://github.org/Juniper/contrail-controller/commit/093321ae25edf3697ff2cc5c1a91418fe8fa9806
Submitter: Zuul
Branch: R3.1

commit 093321ae25edf3697ff2cc5c1a91418fe8fa9806
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:32:17 2016 -0700

Making certfile/keyfile optional, so that vnc_api can rely on CA or CA/CERT.

Change-Id: Iffb9bf9d8cf23fe3943335565bf2adaf878c5df8
Partial-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25066
Committed: http://github.org/Juniper/contrail-controller/commit/e94b27980e0576811e831c6a8789cbd056866dfc
Submitter: Zuul
Branch: master

commit e94b27980e0576811e831c6a8789cbd056866dfc
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:32:17 2016 -0700

Making certfile/keyfile optional, so that vnc_api can rely on CA or CA/CERT.

Change-Id: Iffb9bf9d8cf23fe3943335565bf2adaf878c5df8
Partial-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25068
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/c5811a74b3dab0c2682cfadd1530b8936ea14a91
Submitter: Zuul
Branch: master

commit c5811a74b3dab0c2682cfadd1530b8936ea14a91
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:42:18 2016 -0700

Making certfile/keyfile optional, so that contrail plugin can rely on CA or CA/CERT.

Change-Id: I3cd9656c2665aebaf100947f473a8f01be0bbdb8
Closes-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25071
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/c905cb38e0aa3711e9f101c146b393aac90f858c
Submitter: Zuul
Branch: R3.1

commit c905cb38e0aa3711e9f101c146b393aac90f858c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:42:18 2016 -0700

Making certfile/keyfile optional, so that contrail plugin can rely on CA or CA/CERT.

Change-Id: I3cd9656c2665aebaf100947f473a8f01be0bbdb8
Closes-Bug: 1630513

[Martin Millnert (r-martin-5)]

Ignatious, spot the typo: https://github.com/Juniper/contrail-controller/commit/e94b27980e0576811e831c6a8789cbd056866dfc#diff-7c86bfdc416a185870c50db9633701a0R253

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25297
Submitter: Martin Millnert (<email address hidden>)

[Martin Millnert (r-martin-5)]

Ok - that was unexpectedly matched to this bug rather than the correct bug, https://bugs.launchpad.net/juniperopenstack/+bug/1636550 , due to my bug reference in the commit message. :)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25297
Committed: http://github.org/Juniper/contrail-controller/commit/e30fa40409c3e465ffad6fed12c7dd0e45b46e1c
Submitter: Zuul
Branch: master

commit e30fa40409c3e465ffad6fed12c7dd0e45b46e1c
Author: Martin Millnert <email address hidden>
Date: Tue Oct 25 17:22:14 2016 +0200

Fix variable typo introduced in Bug 1630513

Bug https://bugs.launchpad.net/juniperopenstack/+bug/1630513 (native TLS
support between OpenStack's Neutron and contrail-api's neutronapi),
which is long awaited, introduced a variable typo in
contrail-controller/src/api-lib/vnc_api.py.

This fix fixes the bug, and has been tested working in our dev environment,
in so far as neutron-plugin-contrail has stopped throwing stacktraces due
to non-existing variables.

Signed-off-by: Martin Millnert <email address hidden>

Change-Id: I7757e545b3e9b8d302660d562512d6b897823ea1
Closes-Bug: #1636550

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25070
Committed: http://github.org/Juniper/contrail-controller/commit/d7407a1fbb0876f0a84a0864824b3eb3c6ef591d
Submitter: Zuul
Branch: R3.0.3.x

commit d7407a1fbb0876f0a84a0864824b3eb3c6ef591d
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:32:17 2016 -0700

Making certfile/keyfile optional, so that vnc_api can rely on CA or CA/CERT.

Change-Id: Iffb9bf9d8cf23fe3943335565bf2adaf878c5df8
Partial-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25072
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/bb60e96011934ccb1a90583dcb87e55cb5880c4e
Submitter: Zuul
Branch: R3.0.3.x

commit bb60e96011934ccb1a90583dcb87e55cb5880c4e
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:42:18 2016 -0700

Making certfile/keyfile optional, so that contrail plugin can rely on CA or CA/CERT.

Change-Id: I3cd9656c2665aebaf100947f473a8f01be0bbdb8
Closes-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26345
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26346
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26347
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26345
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26346
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26347
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26345
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26346
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26347
Committed: http://github.org/Juniper/contrail-controller/commit/df192ce6f9623c628dee975754027f827dbc28d9
Submitter: Zuul (<email address hidden>)
Branch: master

commit df192ce6f9623c628dee975754027f827dbc28d9
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Nov 21 15:07:15 2016 -0800

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Closes-Bug: 1639426
Closes-Bug: 1630513

Conflicts:
 src/api-lib/vnc_api.py

Change-Id: Ib5e66bfdd27795bd090c3b3b49207241cbc5f0ae

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26346
Committed: http://github.org/Juniper/contrail-controller/commit/60192ce6c5ca663c4faf8b0f1641a0661d96a6e9
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 60192ce6c5ca663c4faf8b0f1641a0661d96a6e9
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Nov 21 15:07:15 2016 -0800

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Closes-Bug: 1639426
Closes-Bug: 1630513

Conflicts:
 src/api-lib/vnc_api.py

Change-Id: I599389972824c1cad37962306fac023bf16ce91c

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26791
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26792
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26345
Committed: http://github.org/Juniper/contrail-controller/commit/8d5be39d807df9993e84279fe9bf9c409a2dda20
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 8d5be39d807df9993e84279fe9bf9c409a2dda20
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Nov 21 15:07:15 2016 -0800

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Change-Id: Ic4e6da9dbbb2118b840ba7d693bf5ee6803f6b01
Closes-Bug: 1639426
Closes-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26791
Committed: http://github.org/Juniper/contrail-packaging/commit/7acd1ab51453f935d9980e922f3a3cc1eada836d
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7acd1ab51453f935d9980e922f3a3cc1eada836d
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sun Dec 4 21:49:24 2016 -0800

Using python certifi from xenial repo in trusty builds
to get the fix in certifi package to use the ubuntu's
system wide CA's.

Change-Id: I4270b1230550b5fbf718ab01c51ba97e1b2bee2c
Closes-Bug: 1630513

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26792
Committed: http://github.org/Juniper/contrail-packaging/commit/b889baa299c08b3ff7dbb1714a990cb218061efb
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit b889baa299c08b3ff7dbb1714a990cb218061efb
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sun Dec 4 21:49:24 2016 -0800

Using python certifi from xenial repo in trusty builds
to get the fix in certifi package to use the ubuntu's
system wide CA's.

Change-Id: I4270b1230550b5fbf718ab01c51ba97e1b2bee2c
Closes-Bug: 1630513
(cherry picked from commit 7acd1ab51453f935d9980e922f3a3cc1eada836d)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27289
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27291
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27292
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27291
Committed: http://github.org/Juniper/contrail-packaging/commit/5751f791dce07fefeb81da5eacbe09d9fd76ab35
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 5751f791dce07fefeb81da5eacbe09d9fd76ab35
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sun Dec 4 21:49:24 2016 -0800

Using python certifi from xenial repo in trusty builds
to get the fix in certifi package to use the ubuntu's
system wide CA's.

Change-Id: I4270b1230550b5fbf718ab01c51ba97e1b2bee2c
Closes-Bug: 1630513
(cherry picked from commit 7acd1ab51453f935d9980e922f3a3cc1eada836d)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27292
Committed: http://github.org/Juniper/contrail-controller/commit/18a920da6f4ce95a66565a5e61ed9b5d6af39d4f
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 18a920da6f4ce95a66565a5e61ed9b5d6af39d4f
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Nov 21 15:07:15 2016 -0800

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Closes-Bug: 1639426
Closes-Bug: 1630513

Getting certs as argument to the VncApi class and creating
unique certbundle for request to different api-servers.
Closes-Bug: 1644713
Closes-Bug: 1644707

Change-Id: Ib5e66bfdd27795bd090c3b3b49207241cbc5f0ae
(cherry picked from commit df192ce6f9623c628dee975754027f827dbc28d9)
(cherry picked from commit d49aec87815d0b881aaec405832c5ac581e29c3d)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27357
Committed: http://github.org/Juniper/contrail-controller/commit/fa7307e874566ceaf4c083dc82508587de19ed55
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit fa7307e874566ceaf4c083dc82508587de19ed55
Author: Ignatious Johnson Christopher <email address hidden>
Date: Wed Oct 19 12:32:17 2016 -0700

Making certfile/keyfile optional, so that vnc_api can rely on CA or CA/CERT.

Change-Id: Iffb9bf9d8cf23fe3943335565bf2adaf878c5df8
Partial-Bug: 1630513
(cherry picked from commit d7407a1fbb0876f0a84a0864824b3eb3c6ef591d)

Issue:
Password is displayed in the log files of the config daemon, during
uncaught exceptions.

Fix:
cgitb sets sys.excepthook to format uncaught exceptions. Deriving the
cgitb Hook and modifying the handle method to mask password along
with formatting.

Change-Id: I5b4251f2ebe0205465b15430a9ef38ef04b3a634
Closes-Bug: 1626317
(cherry picked from commit 6dc670c851d31b12ffa0f07f418b74705e3b5902)

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Closes-Bug: 1639426
Closes-Bug: 1630513

Getting certs as argument to the VncApi class and creating
unique certbundle for request to different api-servers.
Closes-Bug: 1644713
Closes-Bug: 1644707

Change-Id: Ib5e66bfdd27795bd090c3b3b49207241cbc5f0ae
(cherry picked from commit df192ce6f9623c628dee975754027f827dbc28d9)
(cherry picked from commit d49aec87815d0b881aaec405832c5ac581e29c3d)
(cherry picked from commit 18a920da6f4ce95a66565a5e61ed9b5d6af39d4f)

Conflicts:
 src/api-lib/vnc_api.py

Adding the missing import, due to cherry-pick from
a branch which has import os earlier to commit.

Change-Id: Ibbdf7173ffd30d64526a7ecb525c109ff37098a3
Closes-Bug: 1644707
(cherry picked from commit 6223e65dd1ecda43ab6b686a924eaa5d2ff9c035)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27553
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27553
Committed: http://github.org/Juniper/contrail-packaging/commit/93e03bbbf03ca47d40a26731101187f76c54bc2b
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 93e03bbbf03ca47d40a26731101187f76c54bc2b
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sun Dec 4 21:49:24 2016 -0800

Using python certifi from xenial repo in trusty builds
to get the fix in certifi package to use the ubuntu's
system wide CA's.

Change-Id: I4270b1230550b5fbf718ab01c51ba97e1b2bee2c
Closes-Bug: 1630513
(cherry picked from commit 7acd1ab51453f935d9980e922f3a3cc1eada836d)
(cherry picked from commit b889baa299c08b3ff7dbb1714a990cb218061efb)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27898
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27898
Committed: http://github.org/Juniper/contrail-controller/commit/edeac12c6f0fb44e79039d914da28153fca10cb7
Submitter: Zuul (<email address hidden>)
Branch: R3.1.1.x

commit edeac12c6f0fb44e79039d914da28153fca10cb7
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Nov 21 15:07:15 2016 -0800

Certificates needs to be chanined and bundled
in the order (certfile, keyfile and cacert).

1. Chaining in the certificate in correct order
2. Making certfile/keyfile optional

Closes-Bug: 1639426
Closes-Bug: 1630513

Getting certs as argument to the VncApi class and creating
unique certbundle for request to different api-servers.
Closes-Bug: 1644713
Closes-Bug: 1644707

Change-Id: Ib5e66bfdd27795bd090c3b3b49207241cbc5f0ae
(cherry picked from commit df192ce6f9623c628dee975754027f827dbc28d9)
(cherry picked from commit d49aec87815d0b881aaec405832c5ac581e29c3d)
(cherry picked from commit 18a920da6f4ce95a66565a5e61ed9b5d6af39d4f)

Conflicts:
 src/api-lib/vnc_api.py

When insecure flag is set to True in contrail-keystone-auth.conf,
contrail-api fails to start as the cafile is not initialized.
Initializing cafile to empty string, to handle insecure connections.

Change-Id: I23e4fd8ba533000e041fc892845ccc0bbd50fc48
Closes-Bug: 1650697
(cherry picked from commit 9c6d9ca425e9030fdab01db81f15eac479772854)