Nonpublic deploy ramdisk/kernel images are permitted

[Jay Faulkner]

Currently in Ironic, users are instructed that all images must be publically accessible to be used (visibility=public).

To reproduce:
- Upload kernel/ramdisk images with visibility=shared (default?)
- Run a cleaning (manage/provide) with a clean step that requests a reboot ( https://review.opendev.org/c/openstack/ironic-python-agent/+/941714 )
- The kernel/ramdisk works the first time through (passes service_utils.is_image_available)
- When attempting to reboot, is_image_available appears to reject the image it accepted the first time through.

After both kernel and ramdisk were set to visibility=public, cleaning proceeded properly after the reboot.

I think there's a possibility that the API request to clean a node (baremetal node provide node-0) is somehow passing on a project_id that is acceptable to is_image_available, but then later it fails because we don't have the context with that auth the second time through.

We should have consistent behavior here. It seems wrong if we're checking context.project_id for a ramdisk ever as those are generally not project affiliated (or are they).

I think this bug has potential security ramifications, and so I'm making it a private security bug until more Ironic developers have a chance to ensure I've got the scope of this correct.

[Jay Faulkner (jason-oldos)]

Adding Doug and Satoshi as kinda co-reporters. Satoshi found the issue, Doug helped us narrow it down as he had a similar bug.

[Jay Faulkner (jason-oldos)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jay Faulkner (jason-oldos)]

I am 90% sure this is not severe enough to embargo; but going through the motions until other Ironic cores get a look.

[Jay Faulkner (jason-oldos)]

Looked over this with Julia; there's no security vulnerability here because to get the "first run of cleaning" to work with a private image, you'd need the user who owns the private or shared glance image to be the one to actually call the API to begin cleaning/servicing/etc. This means that implicitly, in order to activate this bug you *must* have already had the rights to see the contents of the image.

This feature was originally intended to allow private images owned *by the user ironic-conductor has credentials for* -- not for the requestor to be able to access/use private images.

So a potential fix here is:
1) Add additional logging in is_image_available (or transmit more detailed status to caller and log there)

2) Add a check to ensure that we don't care about context.project_id in cases of administrative tasks e.g. servicing/cleaning/any ramdisk usage or workflow that Ironic might need to fetch the image more than one time. (default: true, needs to have a CONF knob to reduce risk of regressions)

3) Update the check in is_image_available, if possible, to check for *ironic conductor's* project ID versus the one in the keystone request context. (default: on, needs to be disablable via CONF to reduce risk of aggressions) -- this is going to be significantly more difficult and invasive than steps 1+2 and should be done as a separate change, and likely not backported

As an additional improvement here, we can add support (in a different, non-backportable change), the ability for us to use visibility=shared images. There may already be proposed code to do this.

[Jay Faulkner (jason-oldos)]

With Julia's insight, we're opening this up. It's not a security bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/942641

[Doug Goldstein (cardoe)]

So for the anaconda deploy case, the only thing that needed to be public was the main image. Which isn't even the kernel or the ramdisk but instead is the os-image.tar.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/942641
Committed: https://opendev.org/openstack/ironic/commit/e7d1f8e21114ea742201463b90742757cdaed8df
Submitter: "Zuul (22348)"
Branch: master

commit e7d1f8e21114ea742201463b90742757cdaed8df
Author: Satoshi-Sh <email address hidden>
Date: Mon Feb 24 18:03:49 2025 +0000

    Add extra log to is_image_available

    Added a reason why the image is not available for the debuggin.

    Partial-Bug: #2099276
    Change-Id: Id0f71e201e7e4509e4dd34fa18d1c980dc28b4d3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/943028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/943028
Committed: https://opendev.org/openstack/ironic/commit/1dbb501cd1168ed0798345ee966ffba10298e5b3
Submitter: "Zuul (22348)"
Branch: master

commit 1dbb501cd1168ed0798345ee966ffba10298e5b3
Author: Satoshi-Sh <email address hidden>
Date: Fri Feb 28 15:59:07 2025 +0000

    Add ignore_project_check_for_admin_tasks config option

    Add a new config variable to ignore project_id checks in administrative tasks

    Partial-Bug: #2099276
    Change-Id: I3c6ba8f995a2781229c07c047f66e6737109cdc9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/946575

[Satoshi Shirosaka (satoshi-sh)]

Note for this implementation from the maintainors

  # Order of operations:
    #
    # 1) Patch 1 (backportable):
    # - Add a conf value as a "get the old behavior" escape hatch (deprecated to use this, for removal ???)
    # - make sure escape hatch is default OFF in master, default ON in stable
    # - Add 'community' as a valid image_visibility that is treated identically to 'public'
    # - If that conf is TRUE, do the existing auth_token check.
    # - If that conf is FALSE, do not use existing auth_token check
    # this effectively makes the logic:
    # - If image visibility is public, ALLOW
    # - If image visibility is community, ALLOW
    # - if is_admin & CONF.ignore_project_check_for_admin_tasks=True, ALLOW
    # - If image visibility is private AND image_owner == conductor_project_id, ALLOW
    #
    # MAKE SURE THE BELOW COMMENT EXISTS IN SOME FORM IN PATCH 1 (easy to make bad, insecure assumptions!)
    #
    # NOTE: Any support for private/shared images in Ironic requires a secure
    # way for ironic to know the original requestor:
    # - If we trust node[instance_info][project_id], we are susceptible to a
    # node.owner stealing another project's private image by lying in instance_info.
    # - As of 2025.1, the project_id attached to the auth context at this point
    # is more likely to be the nova-computes service user rather than the original
    # requestor. This is a missing feature from the Ironic/Nova virt driver.
    #
    # 2) Patch 2 (not backportable) - partial shared image support feature
    # - If image visibility is shared, and conductor_project_id in member list, allow
    # - https://docs.openstack.org/api-ref/image/v2/#list-image-members
    #
    # NOTE: The check of `if auth_token` needs to be gone entirely.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/947115

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/946575
Committed: https://opendev.org/openstack/ironic/commit/854f059b82506133d5b0ad37ef57ed4f5635282a
Submitter: "Zuul (22348)"
Branch: master

commit 854f059b82506133d5b0ad37ef57ed4f5635282a
Author: satoshi-sh <email address hidden>
Date: Mon Apr 7 14:11:37 2025 +0000

    Improve is_image_available

    - Add 'community' visiblity alongside 'public'
    - Make the auth token check configurable with a new option
      'allow_image_access_via_auth_token' (default: OFF in
      master, ON is stable)
    - Add testing for is_image_available

    Partial-Bug: #2099276
    Change-Id: I10df3e8fd0091e70f3fb1bc19524aada296c13c1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/ironic/+/947276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/947276
Committed: https://opendev.org/openstack/ironic/commit/d5495ac172653383c9c5b80dc28b3e173cb80d55
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit d5495ac172653383c9c5b80dc28b3e173cb80d55
Author: satoshi-sh <email address hidden>
Date: Mon Apr 7 14:11:37 2025 +0000

    Improve is_image_available

    - Add 'community' visiblity alongside 'public'
    - Make the auth token check configurable with a new option
      'allow_image_access_via_auth_token' (default: OFF in
      master, ON is stable)
    - Add testing for is_image_available

    Note:
    The default value of `allow_image_access_via_auth_token`
    is set to `True` in this stable branch (unlike master
    where it defaults to `False`), to maintain compatibility
    with prior behavior.

    Partial-Bug: #2099276
    Change-Id: I10df3e8fd0091e70f3fb1bc19524aada296c13c1
    (cherry picked from commit 854f059b82506133d5b0ad37ef57ed4f5635282a)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/ironic/+/947432

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/ironic/+/947435

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Satoshi Shirosaka <email address hidden>" on branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/ironic/+/947432
Reason: We're going to release 2025.1 with this fix

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Satoshi Shirosaka <email address hidden>" on branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/ironic/+/947435
Reason: We're going to release 2025.1 with this fix

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/947115
Committed: https://opendev.org/openstack/ironic/commit/a39f11cecee1e0ae7d2ff452da4459cd0f4238d4
Submitter: "Zuul (22348)"
Branch: master

commit a39f11cecee1e0ae7d2ff452da4459cd0f4238d4
Author: Satoshi S. <email address hidden>
Date: Mon Apr 14 12:45:02 2025 +0000

    Add shared image support

    - If image availability is shared and conductor_project_id is in the
      image shared member list, allows access

    Closes-Bug: #2099276
    Change-Id: I6b8a10fe82b41aa37b4f14bca9d3c0c498882bd1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 30.0.0 Flamingo release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ironic/+/968087

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/968087
Committed: https://opendev.org/openstack/ironic/commit/7ce4fdfb100380eddbbddd548e5957c34597d1f0
Submitter: "Zuul (22348)"
Branch: master

commit 7ce4fdfb100380eddbbddd548e5957c34597d1f0
Author: Doug Goldstein <email address hidden>
Date: Fri Nov 21 15:10:22 2025 -0600

    fix: glance image member lookup resulted in an empty list always

    The code did not initialize a client so it resulted in an exception
    always occurring. It also used the image attribute on the client
    attribute of the service but the client attribute of the service is
    already the image attribute. Create a wrapper method to use the API
    correctly and prevent similar issues.

    Closes-Bug: #2099276
    Change-Id: Ib803c066ca28d1c05a345b7a982a0daabbd7d52e
    Signed-off-by: Doug Goldstein <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.2
Review: https://review.opendev.org/c/openstack/ironic/+/968226

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ironic/+/968226
Committed: https://opendev.org/openstack/ironic/commit/6c495a9c76668687093eba4c8eccdfe57cc93655
Submitter: "Zuul (22348)"
Branch: stable/2025.2

commit 6c495a9c76668687093eba4c8eccdfe57cc93655
Author: Doug Goldstein <email address hidden>
Date: Fri Nov 21 15:10:22 2025 -0600

    fix: glance image member lookup resulted in an empty list always

    The code did not initialize a client so it resulted in an exception
    always occurring. It also used the image attribute on the client
    attribute of the service but the client attribute of the service is
    already the image attribute. Create a wrapper method to use the API
    correctly and prevent similar issues.

    Closes-Bug: #2099276
    Closes-Bug: #2132246
    Change-Id: Ib803c066ca28d1c05a345b7a982a0daabbd7d52e
    Signed-off-by: Doug Goldstein <email address hidden>
    (cherry picked from commit 7ce4fdfb100380eddbbddd548e5957c34597d1f0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 33.0.0 Gazpacho release.