[RBD] Cinder started requiring write access to glance images RBD pool

[Radosław Piliszek]

Bug summary:

Cinder started requiring write access to glance images RBD pool (on creating volume from image).
See the log excerpt at the report's bottom for the error message.

Identified cause:

I identified the following change as introducing this issue.
"RBD: use correct stripe unit in clone operation"
Change-Id: Iec111ab83e9ed8182c9679c911e3d90927d5a7c3
https://review.opendev.org/c/openstack/cinder/+/786260

Unfortunately, it has been backported back to Train and broke stable deployments. :-(

The `self.rbd.Image` invocation requires write privileges to the image's pool.
This goes against the recommended caps set: https://docs.ceph.com/en/pacific/rbd/rbd-openstack/
The workaround is to revert the change or temporarily set the write access as allowed for Cinder.

Tested versions:
Cinder Victoria + Ceph Nautilus
Cinder Xena + Ceph Octopus

QA suggestions:

Ensure that cinder backend is tested with the recommended caps (images read-only).
Ensure that a raw image is tested (the default qcow2 image of cirros gets copied, not cloned).

Log excerpt:

2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1032, in _create_from_image
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server image_service)
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.7/site-packages/cinder/volume/drivers/rbd.py", line 1584, in clone_image
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server volume_update = self._clone(volume, pool, image, snapshot)
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.7/site-packages/cinder/volume/drivers/rbd.py", line 1034, in _clone
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server stripe_unit = self._get_stripe_unit(src_client.ioctx, src_image)
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.7/site-packages/cinder/volume/drivers/rbd.py", line 1018, in _get_stripe_unit
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server image = self.rbd.Image(ioctx, volume_name)
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server File "rbd.pyx", line 2960, in rbd.Image.__init__
2021-10-17 19:58:58.820 28 ERROR oslo_messaging.rpc.server rbd.PermissionError: [errno 1] error opening image b'XXX' at snapshot None

[Radosław Piliszek (yoctozepto)]

Marked as confirmed by Kolla CI: https://review.opendev.org/c/openstack/kolla-ansible/+/814324
(results pending but I saw it from Zuul console already)

[Mitchell Walls (miwalls)]

I don't think it is the same thing. I think it is this. https://review.opendev.org/c/openstack/cinder/+/786260

[Mitchell Walls (miwalls)]

Apologies I should've read the whole thing. I can say that giving write permission caps in ceph does not workaround https://review.opendev.org/c/openstack/cinder/+/786260

[Radosław Piliszek (yoctozepto)]

Ah, right, you are hitting the other side of the issue. You are getting hit because of the lack of the original (buggy) fix.
This one is caused by that fix.
I will fix the link.
Your issue is https://bugs.launchpad.net/cinder/+bug/1931004

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/815701

[Javier Cacheiro (javicacheiro)]

We have been also hit this issue.

After upgrading openstack volume from image creation fails. This means new instances also fail.
I have been debugging this issue because it was also affecting us after upgrading openstack and getting the new cinder code that corrects the problem with the stripe unit (in cinder it uses by default 8MB but in glance it uses 4MB).

The problem is that the new fix has an important error because it tries to open the glance image in read-write mode (which is the default in rbd.Image).

So the correct way it would be to open it using:

image = self.rbd.Image(ioctx, volume_name, read_only=True)

You must explicitly set the read_only=True (by default is False).

I have just sent a PR to fix this issue:

https://review.opendev.org/c/openstack/cinder/+/815701

I think the importance should be raised to critical because it will break current installs after upgrading causing a great pain between sysadmins.

[Javier Cacheiro (javicacheiro)]

For affected sysadmins, that are having a hard time, while the PR is not merged and integrated in a release, one possible workaround is to replace rbd.py with the one in the PR and restart cinder_volume.

[Sofia Enriquez (lsofia-enriquez)]

Fix proposed to master https://review.opendev.org/c/openstack/cinder/+/783643/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/783643
Committed: https://opendev.org/openstack/cinder/commit/e644e3584b040823f3b3d9d73730b861a3ec413f
Submitter: "Zuul (22348)"
Branch: master

commit e644e3584b040823f3b3d9d73730b861a3ec413f
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinder/+/816957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/817223

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/816957
Committed: https://opendev.org/openstack/cinder/commit/5b169aeea76baecf0b6795efd09dfb1c58ff20d9
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5b169aeea76baecf0b6795efd09dfb1c58ff20d9
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a
    (cherry picked from commit e644e3584b040823f3b3d9d73730b861a3ec413f)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/817261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/817223
Committed: https://opendev.org/openstack/cinder/commit/f2fe6cc115da08670e7aab39fc21b354ce55d3cd
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f2fe6cc115da08670e7aab39fc21b354ce55d3cd
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a
    (cherry picked from commit e644e3584b040823f3b3d9d73730b861a3ec413f)
    (cherry picked from commit 5b169aeea76baecf0b6795efd09dfb1c58ff20d9)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/817417

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/817261
Committed: https://opendev.org/openstack/cinder/commit/5379af08f0a821f6d4cf1571e83680d55e5c17b8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 5379af08f0a821f6d4cf1571e83680d55e5c17b8
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a
    (cherry picked from commit e644e3584b040823f3b3d9d73730b861a3ec413f)
    (cherry picked from commit 5b169aeea76baecf0b6795efd09dfb1c58ff20d9)
    (cherry picked from commit f2fe6cc115da08670e7aab39fc21b354ce55d3cd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/817417
Committed: https://opendev.org/openstack/cinder/commit/d69f309053bb467e7cb0b10928638d6c381a4b83
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit d69f309053bb467e7cb0b10928638d6c381a4b83
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a
    (cherry picked from commit e644e3584b040823f3b3d9d73730b861a3ec413f)
    (cherry picked from commit 5b169aeea76baecf0b6795efd09dfb1c58ff20d9)
    (cherry picked from commit f2fe6cc115da08670e7aab39fc21b354ce55d3cd)
    (cherry picked from commit 5379af08f0a821f6d4cf1571e83680d55e5c17b8)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Eric Harney <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/815701
Reason: Superseded by https://review.opendev.org/c/openstack/cinder/+/783643/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/cinder/+/817546

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/817546
Committed: https://opendev.org/openstack/cinder/commit/3a7b2e3064319ef3ee844cdebfa9bbcaa4d6b377
Submitter: "Zuul (22348)"
Branch: stable/train

commit 3a7b2e3064319ef3ee844cdebfa9bbcaa4d6b377
Author: Eric Harney <email address hidden>
Date: Mon Mar 29 08:32:58 2021 -0400

    RBD: Open RBD images read-only where possible

    In cases where we don't need to modify the image,
    open rbd images in read-only mode.

    Closes-Bug: #1947518
    Change-Id: I8287460b902dd525aa5313861142f5fb8490e60a
    (cherry picked from commit e644e3584b040823f3b3d9d73730b861a3ec413f)
    (cherry picked from commit 5b169aeea76baecf0b6795efd09dfb1c58ff20d9)
    (cherry picked from commit f2fe6cc115da08670e7aab39fc21b354ce55d3cd)
    (cherry picked from commit 5379af08f0a821f6d4cf1571e83680d55e5c17b8)
    (cherry picked from commit d69f309053bb467e7cb0b10928638d6c381a4b83)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 20.0.0.0rc1 release candidate.

[Christian Rohmann (christian-rohmann)]

Thanks for fixing this! I just ran into this on a Victoria install on Ubuntu Focal and noticed I can only have cinder in version 17.2.0.

When will 17.3.0 be built and made available via Ubuntu Cloud Archive? - https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/victoria_versions.html still shows 17.2.0 with 17.3.0 no even in staging ...

[Christian Rohmann (christian-rohmann)]

While this is fixed for Victoria with 17.3, there are are no packages provided by Cloud Archive yet - https://bugs.launchpad.net/cloud-archive/+bug/1947518/comments/27

[Corey Bryant (corey.bryant)]

For Ubuntu:
Xena: cinder 19.1.0 will be tracked in https://bugs.launchpad.net/cloud-archive/+bug/1972665
Victoria: cinder 17.4.0 will be tracked in https://bugs.launchpad.net/cloud-archive/+bug/1976543

[Corey Bryant (corey.bryant)]

Just circling back to this to note that cinder 19.1.0 and 17.4.0 have been released for the ubuntu cloud archive.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder train-eol release.