libvirt accessible by default via an unauthenticated, unencrypted TCP connection

[Mark Goddard]

In Kolla Ansible OpenStack deployments, by default, libvirt is configured to allow read-write access via an unauthenticated, unencrypted TCP connection, using the internal API network. This is to facilitate migration between hosts.

By default, Kolla Ansible does not use encryption for services on the internal network (and did not support it until Ussuri). However, most other services on the internal network are at least authenticated (usually via passwords), ensuring that they cannot be used by anyone with access to the network, unless they have credentials.

The main issue here is the lack of authentication. Any client with access to the internal network is able to connect to the libvirt TCP port and make arbitrary changes to the hypervisor. This could include starting a VM, modifying an existing VM, etc. Given the flexibility of the domain options, it could be seen as equivalent to having root access to the hypervisor.

Kolla Ansible supports libvirt TLS [1] since the Train release, using client and server certificates for mutual authentication and encryption. However, this feature is not enabled by default, and requires certificates to be generated for each compute host.

Kolla Ansible should configure SASL username/password authentication by default, as described in https://libvirt.org/auth.html, to provide a base level of security. Deployments requiring further security should use libvirt TLS.

[1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

[Mark Goddard (mgoddard)]

This bug report suggests libvirt client certs alone are not secure as an authentication method. https://bugzilla.redhat.com/show_bug.cgi?id=1513440. Unfortunately I can't access the linked bug report, https://bugzilla.redhat.com/show_bug.cgi?id=1510216.

[Radosław Piliszek (yoctozepto)]

Argh, it's so unwieldy to try to review this without Gerrit. I suggest we move there, it's not that critical to keep hiding.

https://access.redhat.com/blogs/766093/posts/1976653

That said, here are my 2 cents:

1) Kolla:

a) why add MD5? it's insecure anyway, scram should be enough
b) that package does not exist in Debuntu - need to use libsasl2-modules-gssapi-mit

2) Kolla Ansible:

a) the doc says libvirt tls uses mutual tls for auth (which is true) and disables sasl - this is wrongas you enable scram if tls and md5 if not, never disabling sasl - I suggest we disable sasl indeed
b) that same doc tells the truth slightly above the other remark - please explain
c) why not enable scram without tls too?

[Mark Goddard (mgoddard)]

Thanks for checking out the patch. I was simply following the process here: https://security.openstack.org/#openstack-security-notes

Let's discuss the approach here, then move to gerrit to nitpick the implementation.

1)

a) I agree this is not ideal, but SCRAM is not supported on TCP connections. My patch uses SCRAM with TLS.

b) I was looking at https://packages.ubuntu.com/source/focal/cyrus-sasl2, but it's a source package. I guess we want sasl2-bin. libsasl2-modules-gssapi-mit looks like it's for GSSAPI, although does pull in libscram. I'll try those two.

2)

a) b) See comment #1. I switched to enabling it unconditionally after reading that. It shouldn't hurt, and can be disabled if necessary. I can tidy up inconsistencies in the patch.

c) See 1) a).

[Radosław Piliszek (yoctozepto)]

Duh that it's not supported! It does not make sense to support MD5 in 2022 but let it be.
Fix packages and docs and let's go go go.

[Radosław Piliszek (yoctozepto)]

Maybe we don't need to care about SCRAM then? I would not bother with SASL and TLS.

[Radosław Piliszek (yoctozepto)]

Regarding the process - fungi told me not to be too strict with that and gave me the link I posted. Unless the issue is critical (and, as I understand, kind of brutal 0-day), there is no need to hide the fixing process.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/833022
Committed: https://opendev.org/openstack/kolla-ansible/commit/d2d4b53d47df3b1a250c21404a8ec140873d4ce5
Submitter: "Zuul (22348)"
Branch: master

commit d2d4b53d47df3b1a250c21404a8ec140873d4ce5
Author: Mark Goddard <email address hidden>
Date: Fri Mar 4 17:57:24 2022 +0000

    libvirt: support SASL authentication

    In Kolla Ansible OpenStack deployments, by default, libvirt is
    configured to allow read-write access via an unauthenticated,
    unencrypted TCP connection, using the internal API network. This is to
    facilitate migration between hosts.

    By default, Kolla Ansible does not use encryption for services on the
    internal network (and did not support it until Ussuri). However, most
    other services on the internal network are at least authenticated
    (usually via passwords), ensuring that they cannot be used by anyone
    with access to the network, unless they have credentials.

    The main issue here is the lack of authentication. Any client with
    access to the internal network is able to connect to the libvirt TCP
    port and make arbitrary changes to the hypervisor. This could include
    starting a VM, modifying an existing VM, etc. Given the flexibility of
    the domain options, it could be seen as equivalent to having root access
    to the hypervisor.

    Kolla Ansible supports libvirt TLS [1] since the Train release, using
    client and server certificates for mutual authentication and encryption.
    However, this feature is not enabled by default, and requires
    certificates to be generated for each compute host.

    This change adds support for libvirt SASL authentication, and enables it
    by default. This provides base level of security. Deployments requiring
    further security should use libvirt TLS.

    [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

    Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
    Closes-Bug: #1964013
    Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833422

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833423

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833424

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/833422
Committed: https://opendev.org/openstack/kolla-ansible/commit/af6b3edfa032fac7347e3e056ddfea4a539096d5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit af6b3edfa032fac7347e3e056ddfea4a539096d5
Author: Mark Goddard <email address hidden>
Date: Fri Mar 4 17:57:24 2022 +0000

    libvirt: support SASL authentication

    In Kolla Ansible OpenStack deployments, by default, libvirt is
    configured to allow read-write access via an unauthenticated,
    unencrypted TCP connection, using the internal API network. This is to
    facilitate migration between hosts.

    By default, Kolla Ansible does not use encryption for services on the
    internal network (and did not support it until Ussuri). However, most
    other services on the internal network are at least authenticated
    (usually via passwords), ensuring that they cannot be used by anyone
    with access to the network, unless they have credentials.

    The main issue here is the lack of authentication. Any client with
    access to the internal network is able to connect to the libvirt TCP
    port and make arbitrary changes to the hypervisor. This could include
    starting a VM, modifying an existing VM, etc. Given the flexibility of
    the domain options, it could be seen as equivalent to having root access
    to the hypervisor.

    Kolla Ansible supports libvirt TLS [1] since the Train release, using
    client and server certificates for mutual authentication and encryption.
    However, this feature is not enabled by default, and requires
    certificates to be generated for each compute host.

    This change adds support for libvirt SASL authentication, and enables it
    by default. This provides base level of security. Deployments requiring
    further security should use libvirt TLS.

    [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

    Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
    Closes-Bug: #1964013
    Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e
    (cherry picked from commit d2d4b53d47df3b1a250c21404a8ec140873d4ce5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/833423
Committed: https://opendev.org/openstack/kolla-ansible/commit/536ffc3f780138077cf179602bb0da323c368ba3
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 536ffc3f780138077cf179602bb0da323c368ba3
Author: Mark Goddard <email address hidden>
Date: Fri Mar 4 17:57:24 2022 +0000

    libvirt: support SASL authentication

    In Kolla Ansible OpenStack deployments, by default, libvirt is
    configured to allow read-write access via an unauthenticated,
    unencrypted TCP connection, using the internal API network. This is to
    facilitate migration between hosts.

    By default, Kolla Ansible does not use encryption for services on the
    internal network (and did not support it until Ussuri). However, most
    other services on the internal network are at least authenticated
    (usually via passwords), ensuring that they cannot be used by anyone
    with access to the network, unless they have credentials.

    The main issue here is the lack of authentication. Any client with
    access to the internal network is able to connect to the libvirt TCP
    port and make arbitrary changes to the hypervisor. This could include
    starting a VM, modifying an existing VM, etc. Given the flexibility of
    the domain options, it could be seen as equivalent to having root access
    to the hypervisor.

    Kolla Ansible supports libvirt TLS [1] since the Train release, using
    client and server certificates for mutual authentication and encryption.
    However, this feature is not enabled by default, and requires
    certificates to be generated for each compute host.

    This change adds support for libvirt SASL authentication, and enables it
    by default. This provides base level of security. Deployments requiring
    further security should use libvirt TLS.

    [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

    Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
    Closes-Bug: #1964013
    Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e
    (cherry picked from commit d2d4b53d47df3b1a250c21404a8ec140873d4ce5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/833424
Committed: https://opendev.org/openstack/kolla-ansible/commit/8c0709f93cbc6e0011cabdbad10f9d162c93ee32
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8c0709f93cbc6e0011cabdbad10f9d162c93ee32
Author: Mark Goddard <email address hidden>
Date: Fri Mar 4 17:57:24 2022 +0000

    libvirt: support SASL authentication

    In Kolla Ansible OpenStack deployments, by default, libvirt is
    configured to allow read-write access via an unauthenticated,
    unencrypted TCP connection, using the internal API network. This is to
    facilitate migration between hosts.

    By default, Kolla Ansible does not use encryption for services on the
    internal network (and did not support it until Ussuri). However, most
    other services on the internal network are at least authenticated
    (usually via passwords), ensuring that they cannot be used by anyone
    with access to the network, unless they have credentials.

    The main issue here is the lack of authentication. Any client with
    access to the internal network is able to connect to the libvirt TCP
    port and make arbitrary changes to the hypervisor. This could include
    starting a VM, modifying an existing VM, etc. Given the flexibility of
    the domain options, it could be seen as equivalent to having root access
    to the hypervisor.

    Kolla Ansible supports libvirt TLS [1] since the Train release, using
    client and server certificates for mutual authentication and encryption.
    However, this feature is not enabled by default, and requires
    certificates to be generated for each compute host.

    This change adds support for libvirt SASL authentication, and enables it
    by default. This provides base level of security. Deployments requiring
    further security should use libvirt TLS.

    [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

    Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
    Closes-Bug: #1964013
    Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e
    (cherry picked from commit d2d4b53d47df3b1a250c21404a8ec140873d4ce5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/833421
Committed: https://opendev.org/openstack/kolla-ansible/commit/daef31a420a265f0e795f92e824e2093c80edfec
Submitter: "Zuul (22348)"
Branch: stable/xena

commit daef31a420a265f0e795f92e824e2093c80edfec
Author: Mark Goddard <email address hidden>
Date: Fri Mar 4 17:57:24 2022 +0000

    libvirt: support SASL authentication

    In Kolla Ansible OpenStack deployments, by default, libvirt is
    configured to allow read-write access via an unauthenticated,
    unencrypted TCP connection, using the internal API network. This is to
    facilitate migration between hosts.

    By default, Kolla Ansible does not use encryption for services on the
    internal network (and did not support it until Ussuri). However, most
    other services on the internal network are at least authenticated
    (usually via passwords), ensuring that they cannot be used by anyone
    with access to the network, unless they have credentials.

    The main issue here is the lack of authentication. Any client with
    access to the internal network is able to connect to the libvirt TCP
    port and make arbitrary changes to the hypervisor. This could include
    starting a VM, modifying an existing VM, etc. Given the flexibility of
    the domain options, it could be seen as equivalent to having root access
    to the hypervisor.

    Kolla Ansible supports libvirt TLS [1] since the Train release, using
    client and server certificates for mutual authentication and encryption.
    However, this feature is not enabled by default, and requires
    certificates to be generated for each compute host.

    This change adds support for libvirt SASL authentication, and enables it
    by default. This provides base level of security. Deployments requiring
    further security should use libvirt TLS.

    [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls

    Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
    Closes-Bug: #1964013
    Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e
    (cherry picked from commit d2d4b53d47df3b1a250c21404a8ec140873d4ce5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 11.4.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/841660

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Maksim Malchuk <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/841660
Reason: switch to https://review.opendev.org/c/openstack/kolla-ansible/+/834459

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/833425
Reason: train is going eol

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible ussuri-eol release.