neutron api tls issues

[Michele Baldessari]

There are two tls issues currently in neutron:
A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
"Source": "/etc/pki/tls/private/ovn_neutron_client.key",
"Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

"Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
"Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

And then it thinks it can chown it inside the container via the kolla script:
[root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
...
        {
            "optional": true,
            "owner": "neutron:neutron",
            "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
            "perm": "0644"
        },
        {
            "optional": true,
            "owner": "neutron:neutron",
            "path": "/etc/pki/tls/private/ovn_neutron_client.key",
            "perm": "0644"
        }

What needs to happen here is that we drop the bind mounts, and copy in the certs via kolla and put them in /var/lib/config-data/puppet-generated/...... Just like galera is doing.

This typically breaks the updating of the certs, but that is currently not working for neutron_ovn anyway since the cert has no post-command in certmonger:
Request ID 'neutron_ovn':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/ovn_neutron_client.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/ovn_neutron_client.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=REDHAT.LOCAL
        subject: CN=controller-1.internalapi.redhat.local,O=REDHAT.LOCAL
        expires: 2021-11-14 15:44:20 UTC
        dns: controller-1.internalapi.redhat.local
        principal name: <email address hidden>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Also fun fact, it would not even work with bind mounts because a new cert would create a new inode and when you bind mount a single file and change the inode on the host the container will not see any changes at all.

B) The logging in the neutron_server_tls_proxy container is not persisted to the host

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694614

[Michele Baldessari (michele)]

First seen via https://bugzilla.redhat.com/show_bug.cgi?id=1771807

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694769

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kamil Sambor (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/694769

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694614
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a7db39af7cf779e5e236c6f9b54c260feba24b98
Submitter: Zuul
Branch: master

commit a7db39af7cf779e5e236c6f9b54c260feba24b98
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS issues

    There are two tls issues currently in neutron:
    A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
    "Source": "/etc/pki/tls/private/ovn_neutron_client.key",
    "Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

    "Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
    "Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

    And then it thinks it can chown it inside the container via the kolla script:
    [root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
    ...
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
                "perm": "0644"
            },
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/private/ovn_neutron_client.key",
                "perm": "0644"
            }

    What needs to happen here is that we drop the direct bind mounts, and copy in the certs via kolla. Just like galera is doing.

    Tested and with this patch I correctly get a Train/OSP16 TLS everywhere
    deployment with a working neutron:
    Before:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

    After:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    (overcloud) [stack@undercloud-0 ~]$

    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    Change-Id: Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694769
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=491ffccbb99b6d6600e463991f20864cd1eb100e
Submitter: Zuul
Branch: stable/train

commit 491ffccbb99b6d6600e463991f20864cd1eb100e
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS issues

    There are two tls issues currently in neutron:
    A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
    "Source": "/etc/pki/tls/private/ovn_neutron_client.key",
    "Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

    "Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
    "Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

    And then it thinks it can chown it inside the container via the kolla script:
    [root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
    ...
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
                "perm": "0644"
            },
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/private/ovn_neutron_client.key",
                "perm": "0644"
            }

    What needs to happen here is that we drop the direct bind mounts, and copy in the certs via kolla. Just like galera is doing.

    Tested and with this patch I correctly get a Train/OSP16 TLS everywhere
    deployment with a working neutron:
    Before:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

    After:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    (overcloud) [stack@undercloud-0 ~]$

    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    Change-Id: Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779
    (cherry picked from commit a7db39af7cf779e5e236c6f9b54c260feba24b98)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/696652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696652
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0aecbc772384ae4623fe2013c1698825f268e16b
Submitter: Zuul
Branch: stable/stein

commit 0aecbc772384ae4623fe2013c1698825f268e16b
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697334

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697334
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0aaae842beee2468fb726aa92faa8b53d7266a9c
Submitter: Zuul
Branch: stable/rocky

commit 0aaae842beee2468fb726aa92faa8b53d7266a9c
Author: Michele Baldessari <email address hidden>
Date: Wed Dec 4 18:51:43 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    (cherry-picked from 0aecbc772384ae4623fe2013c1698825f268e16b)
    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/699356

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699356
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3c1f10d331444875b3a9bbfa5c052e3ecb124d2d
Submitter: Zuul
Branch: stable/queens

commit 3c1f10d331444875b3a9bbfa5c052e3ecb124d2d
Author: Michele Baldessari <email address hidden>
Date: Wed Dec 4 18:51:43 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    (cherry-picked from 0aecbc772384ae4623fe2013c1698825f268e16b)
    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560
    (cherry picked from commit 0aaae842beee2468fb726aa92faa8b53d7266a9c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.