Apparmor does not allow nova-compute process to access /etc/ssh/ssh_config.d

[DUFOUR Olivier]

In some occurences, an user might want to put some custom ssh configuration in /etc/ssh/ssh_config.d .
In the triggered case, we tried to apply a workaround to an issue with ssh host keys between nodes on different network spaces.
But ultimately, there could be other reason for a user to put a custom configuration in "/etc/ssh/ssh_config.d/" directory, for the very same reason it already allowed by this apparmor rule to read "/etc/ssh/ssh_config".

This was discovered because due to a bug with LP#1969971, a workaround in /etc/ssh/ssh_config.d/ was applied to be able to migrate instances. However for some specific actions, such as resizing an instance, it still might fail.

This was discovered on the following environment :
* Ubuntu Jammy 22.04
* Juju 2.9.45
* Openstack Yoga

The steps to trigger the issue :
* have nova-computes to use different spaces between management and migration network (related to https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1969971/comments/9 )
* apply a workaround to circumvent the issue in LP#1969971 in /etc/ssh/ssh_config.d/
* have nova-compute aa-profile-mode set to "enforce"
* try to resize an existing instance to a different flavor

At a first glance in this situation, there is no easy workaround aside from disabling apparmor, which kind of defeat its purpose.

Due to customer's environment restriction, I'm able to post only screenshot of the logs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902047

[DUFOUR Olivier (odufourc)]

Since the change required to make it work is quite minimal, I have prepared a patch against the charm to allow nova-compute process to access /etc/ssh/ssh_config.d/

This is confirmed to work at least in my lab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/902047
Committed: https://opendev.org/openstack/charm-nova-compute/commit/4d6f4c07c9b634e22d5445a702be3d3ee9730ab3
Submitter: "Zuul (22348)"
Branch: master

commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902525

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902526

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902527

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902528

[DUFOUR Olivier (odufourc)]

Since it impacts existing deployments on Yoga, and the fix now merged against master's branch.
Let's backport the fix from Master branch up to stable/Yoga branch including intermediary releases of Openstack.

[DUFOUR Olivier (odufourc)]

Subscribed Field high

It impacts a customer, and probably many others.

CI is currently broken after "cherry-picking" my fix on top of other branches as requested and I sadly don't have the time on my side to continue working on it.

[Alex Kavanagh (ajkavanagh)]

Olivier, I don't think it's strictly correct to subscribe field-high to this bug, as it's largely a feature addition to enable new functionality. The referenced bug (#1969971) has also got a fix committed to the master branch and will most likely be backported by SEG soon. Please could you re-consider as field-high should not be used for:

> Missing functionality: A field deployment suddenly found to require functionality that does not exist does not qualify for the Field SLA. Please raise the issue with the appropriate engineering manager to determine what can be done.

Thanks.

[Nobuto Murata (nobuto)]

@ajkavanagh, I get your point. However, we obviously didn't intend this to be a new feature but a mitigation to ongoing issues.

Overall, the timeline matters. If the fix for LP: #1969971 gets backported down to Yoga within a few business days, sure we can drop the priority of this. However, the backport hasn't been started and I'm not sure how much risk is there in the backport when changing the way of managing known_hosts.

Olivier is concluding his onsite activity on Dec 12th (Tue) in the APAC timezones so we need something working by the end of Dec 11th (Mon) in other timezones and I thought Olivier's patch would be more suitable to backport in a hurry since it has the minimal risk of regressions.

Does this make sense?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/902525
Committed: https://opendev.org/openstack/charm-nova-compute/commit/fbdb7c9702e890da78dbd8252756f2e6ea843d77
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit fbdb7c9702e890da78dbd8252756f2e6ea843d77
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/902526
Committed: https://opendev.org/openstack/charm-nova-compute/commit/56c8b59192adab4c56d2cd863e34ebe0004ef539
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 56c8b59192adab4c56d2cd863e34ebe0004ef539
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/902527
Committed: https://opendev.org/openstack/charm-nova-compute/commit/dffd4dde74ac1ed4ffa151d0ab0ca99b1590fe56
Submitter: "Zuul (22348)"
Branch: stable/zed

commit dffd4dde74ac1ed4ffa151d0ab0ca99b1590fe56
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/902528
Committed: https://opendev.org/openstack/charm-nova-compute/commit/20c9776e8836abbe13a2eab2efe1d8c9a8cff851
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 20c9776e8836abbe13a2eab2efe1d8c9a8cff851
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)

[DUFOUR Olivier (odufourc)]

Unsubscribed Field high

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/914204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/914204
Committed: https://opendev.org/openstack/charm-nova-compute/commit/552a84925c9c1f7adc810e338a61944b6a53dbd0
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 552a84925c9c1f7adc810e338a61944b6a53dbd0
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)
    (cherry picked from commit 20c9776e8836abbe13a2eab2efe1d8c9a8cff851)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/914454

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/914454
Committed: https://opendev.org/openstack/charm-nova-compute/commit/decbbd95330864a18dfd6d68b84860a9a671cf2b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit decbbd95330864a18dfd6d68b84860a9a671cf2b
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)
    (cherry picked from commit 20c9776e8836abbe13a2eab2efe1d8c9a8cff851)
    (cherry picked from commit 552a84925c9c1f7adc810e338a61944b6a53dbd0)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/914676

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/914676
Committed: https://opendev.org/openstack/charm-nova-compute/commit/c7014adcc5cc25994065887122e5168c1c7c0f6b
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c7014adcc5cc25994065887122e5168c1c7c0f6b
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)
    (cherry picked from commit 20c9776e8836abbe13a2eab2efe1d8c9a8cff851)
    (cherry picked from commit 552a84925c9c1f7adc810e338a61944b6a53dbd0)
    (cherry picked from commit decbbd95330864a18dfd6d68b84860a9a671cf2b)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/915007

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/915007
Committed: https://opendev.org/openstack/charm-nova-compute/commit/a5f4e72933f52e640797ca1c6a1fa9134b345c87
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a5f4e72933f52e640797ca1c6a1fa9134b345c87
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Nov 28 16:16:14 2023 +0900

    Update apparmor profile for nova-compute

    Nova-compute uses ssh and scp commands extensively and this
    patch allows the process to read the configuration too in
    /etc/ssh/ssh_config.d/ directory.

    Closes-Bug: #2044983
    Change-Id: I336ce64d493c549096d0b8706996e0f17a2728fb
    (cherry picked from commit 4d6f4c07c9b634e22d5445a702be3d3ee9730ab3)
    (cherry picked from commit 20c9776e8836abbe13a2eab2efe1d8c9a8cff851)
    (cherry picked from commit 552a84925c9c1f7adc810e338a61944b6a53dbd0)
    (cherry picked from commit decbbd95330864a18dfd6d68b84860a9a671cf2b)
    (cherry picked from commit c7014adcc5cc25994065887122e5168c1c7c0f6b)