Activity log for bug #1900775

Date Who What changed Old value New value Message
2020-10-20 22:37:32 Andrey Grebennikov bug added bug
2020-10-20 22:51:34 Vern Hart bug added subscriber Canonical Field Critical
2020-10-21 00:36:23 Michael Iatrou bug added subscriber Michael Iatrou
2020-10-21 13:13:32 Aurelien Lourot charm-ceph-rbd-mirror: assignee Aurelien Lourot (aurelien-lourot)
2020-10-21 13:34:16 Corey Bryant bug task added cinder
2020-10-21 14:50:09 Aurelien Lourot charm-ceph-rbd-mirror: status New Invalid
2020-10-21 14:50:13 Aurelien Lourot charm-ceph-rbd-mirror: assignee Aurelien Lourot (aurelien-lourot)
2020-10-22 15:30:24 Corey Bryant bug task added cinder (Ubuntu)
2020-10-22 18:55:54 OpenStack Infra cinder: status New In Progress
2020-10-22 18:55:54 OpenStack Infra cinder: assignee Corey Bryant (corey.bryant)
2020-10-22 19:03:17 Corey Bryant nominated for series Ubuntu Groovy
2020-10-22 19:03:17 Corey Bryant bug task added cinder (Ubuntu Groovy)
2020-10-22 19:03:17 Corey Bryant nominated for series Ubuntu Xenial
2020-10-22 19:03:17 Corey Bryant bug task added cinder (Ubuntu Xenial)
2020-10-22 19:03:17 Corey Bryant nominated for series Ubuntu Bionic
2020-10-22 19:03:17 Corey Bryant bug task added cinder (Ubuntu Bionic)
2020-10-22 19:03:17 Corey Bryant nominated for series Ubuntu Focal
2020-10-22 19:03:17 Corey Bryant bug task added cinder (Ubuntu Focal)
2020-10-22 19:03:32 Corey Bryant cinder (Ubuntu Groovy): status New Triaged
2020-10-22 19:03:34 Corey Bryant cinder (Ubuntu Focal): status New Triaged
2020-10-22 19:03:36 Corey Bryant cinder (Ubuntu Bionic): status New Triaged
2020-10-22 19:03:38 Corey Bryant cinder (Ubuntu Xenial): status New Triaged
2020-10-22 19:03:41 Corey Bryant cinder (Ubuntu Xenial): importance Undecided High
2020-10-22 19:03:44 Corey Bryant cinder (Ubuntu Focal): importance Undecided High
2020-10-22 19:03:46 Corey Bryant cinder (Ubuntu Groovy): importance Undecided High
2020-10-22 19:03:50 Corey Bryant cinder (Ubuntu Bionic): importance Undecided High
2020-10-22 19:03:58 Corey Bryant bug task added cloud-archive
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/mitaka
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/mitaka
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/queens
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/queens
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/stein
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/stein
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/victoria
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/victoria
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/train
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/train
2020-10-22 19:04:31 Corey Bryant nominated for series cloud-archive/ussuri
2020-10-22 19:04:31 Corey Bryant bug task added cloud-archive/ussuri
2020-10-22 19:04:46 Corey Bryant cloud-archive/mitaka: importance Undecided High
2020-10-22 19:04:46 Corey Bryant cloud-archive/mitaka: status New Triaged
2020-10-22 19:04:59 Corey Bryant cloud-archive/queens: importance Undecided High
2020-10-22 19:04:59 Corey Bryant cloud-archive/queens: status New Triaged
2020-10-22 19:05:12 Corey Bryant cloud-archive/stein: importance Undecided High
2020-10-22 19:05:12 Corey Bryant cloud-archive/stein: status New Triaged
2020-10-22 19:05:27 Corey Bryant cloud-archive/train: importance Undecided High
2020-10-22 19:05:27 Corey Bryant cloud-archive/train: status New Triaged
2020-10-22 19:05:42 Corey Bryant cloud-archive/ussuri: importance Undecided High
2020-10-22 19:05:42 Corey Bryant cloud-archive/ussuri: status New Triaged
2020-10-22 19:05:53 Corey Bryant cloud-archive/victoria: importance Undecided High
2020-10-22 19:05:53 Corey Bryant cloud-archive/victoria: status New Triaged
2020-10-29 15:33:22 Michael Skalka removed subscriber Canonical Field Critical
2020-10-29 15:33:35 Michael Skalka bug added subscriber Canonical Field High
2021-01-06 19:37:11 Corey Bryant cinder (Ubuntu): status Triaged Fix Released
2021-01-28 16:28:17 Corey Bryant description OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Impact | Test Case] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607
2021-01-28 18:48:22 Corey Bryant description [Impact | Test Case] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607 [Impact] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Test Case] This is a light-weight test to ensure the code is working as expected, using the unit test from the patch: lxc launch ubuntu-daily:hirsute h1 lxc exec h1 /bin/bash root@h1:~# sudo apt install python3-cinder root@h1:~# cd /usr/lib/python3/dist-packages/ /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade self._legacy_facade = LegacyEngineFacade(None, _factory=self) . ---------------------------------------------------------------------- Ran 1 test in 0.701s OK The test will fail if the fixed code is not installed. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607
2021-01-28 18:48:38 Corey Bryant description [Impact] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Test Case] This is a light-weight test to ensure the code is working as expected, using the unit test from the patch: lxc launch ubuntu-daily:hirsute h1 lxc exec h1 /bin/bash root@h1:~# sudo apt install python3-cinder root@h1:~# cd /usr/lib/python3/dist-packages/ /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade self._legacy_facade = LegacyEngineFacade(None, _factory=self) . ---------------------------------------------------------------------- Ran 1 test in 0.701s OK The test will fail if the fixed code is not installed. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607 [Impact] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Test Case] This is a light-weight test to ensure the code is working as expected, using the unit test from the patch: lxc launch ubuntu-daily:hirsute h1 lxc exec h1 /bin/bash root@h1:~# sudo apt install python3-cinder root@h1:~# cd /usr/lib/python3/dist-packages/ /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade self._legacy_facade = LegacyEngineFacade(None, _factory=self) . ---------------------------------------------------------------------- Ran 1 test in 0.701s OK The test will fail if the fixed code is not installed. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607
2021-01-28 18:57:25 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2021-02-10 14:55:00 Corey Bryant cloud-archive/train: status Triaged Fix Committed
2021-02-10 14:55:04 Corey Bryant tags field-critical field-critical verification-train-needed
2021-02-10 14:55:26 Corey Bryant cloud-archive/stein: status Triaged Fix Committed
2021-02-10 14:55:28 Corey Bryant tags field-critical verification-train-needed field-critical verification-stein-needed verification-train-needed
2021-02-16 17:48:04 Brian Murray cinder (Ubuntu Groovy): status Triaged Fix Committed
2021-02-16 17:48:09 Brian Murray bug added subscriber SRU Verification
2021-02-16 17:48:15 Brian Murray tags field-critical verification-stein-needed verification-train-needed field-critical verification-needed verification-needed-groovy verification-stein-needed verification-train-needed
2021-02-16 17:53:45 Brian Murray cinder (Ubuntu Focal): status Triaged Fix Committed
2021-02-16 17:53:52 Brian Murray tags field-critical verification-needed verification-needed-groovy verification-stein-needed verification-train-needed field-critical verification-needed verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed
2021-02-16 17:55:50 Brian Murray cinder (Ubuntu Bionic): status Triaged Fix Committed
2021-02-16 17:55:58 Brian Murray tags field-critical verification-needed verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed
2021-02-16 20:59:04 Corey Bryant cloud-archive/victoria: status Triaged Fix Committed
2021-02-16 20:59:07 Corey Bryant tags field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed verification-victoria-needed
2021-02-16 21:39:12 Corey Bryant cloud-archive/queens: status Triaged Fix Committed
2021-02-16 21:39:14 Corey Bryant tags field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed verification-victoria-needed field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-queens-needed verification-stein-needed verification-train-needed verification-victoria-needed
2021-02-17 07:20:24 Chris MacNaughton cloud-archive/victoria: status Fix Committed Fix Released
2021-02-17 12:53:34 Corey Bryant cloud-archive/ussuri: status Triaged Fix Committed
2021-02-17 12:53:37 Corey Bryant tags field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-queens-needed verification-stein-needed verification-train-needed verification-victoria-needed field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-queens-needed verification-stein-needed verification-train-needed verification-ussuri-needed verification-victoria-needed
2021-02-24 18:37:29 Corey Bryant description [Impact] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Test Case] This is a light-weight test to ensure the code is working as expected, using the unit test from the patch: lxc launch ubuntu-daily:hirsute h1 lxc exec h1 /bin/bash root@h1:~# sudo apt install python3-cinder root@h1:~# cd /usr/lib/python3/dist-packages/ /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade self._legacy_facade = LegacyEngineFacade(None, _factory=self) . ---------------------------------------------------------------------- Ran 1 test in 0.701s OK The test will fail if the fixed code is not installed. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607 [Impact] OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way. Cinder has Ceph as backend for volumes. Creating a volume from qcow2 image. Current flow is the following: 1. Cinder creates empty volume in Ceph 2. Cinder downloads the image 3. Image is converted to raw 4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611 5. Cinder performs "rbd import" using unpacked raw image as source. Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second. It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error. The only way to fix this behaviour - disable mirroring of the backend pool which is not desired. [Test Case] This is a light-weight test to ensure the code is working as expected, using the unit test from the patch: lxc launch ubuntu-daily:hirsute h1 lxc exec h1 /bin/bash root@h1:~# sudo apt install python3-cinder root@h1:~# cd /usr/lib/python3/dist-packages/ root@h1:~# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade   self._legacy_facade = LegacyEngineFacade(None, _factory=self) . ---------------------------------------------------------------------- Ran 1 test in 0.701s OK The test will fail if the fixed code is not installed. [Regression Potential] This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered. [Discussion] This is accompanied by a unit test fix for: https://pad.lv/1913607
2021-02-24 19:19:16 Corey Bryant tags field-critical verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-queens-needed verification-stein-needed verification-train-needed verification-ussuri-needed verification-victoria-needed field-critical verification-done verification-done-bionic verification-done-focal verification-done-groovy verification-queens-done verification-stein-done verification-train-done verification-ussuri-done verification-victoria-done
2021-03-02 22:54:13 Launchpad Janitor cinder (Ubuntu Groovy): status Fix Committed Fix Released
2021-03-02 22:54:54 Launchpad Janitor cinder (Ubuntu Focal): status Fix Committed Fix Released
2021-03-02 22:56:26 Launchpad Janitor cinder (Ubuntu Bionic): status Fix Committed Fix Released
2021-03-03 14:36:17 Corey Bryant cloud-archive/stein: status Fix Committed Fix Released
2021-03-03 14:36:24 Corey Bryant cloud-archive/ussuri: status Fix Committed Fix Released
2021-03-03 14:36:35 Corey Bryant cloud-archive/train: status Fix Committed Fix Released
2021-03-03 14:36:41 Corey Bryant cloud-archive/queens: status Fix Committed Fix Released
2021-03-11 22:02:31 Billy Olsen cloud-archive/mitaka: status Triaged Won't Fix
2021-03-11 22:03:09 Billy Olsen cinder (Ubuntu Xenial): status Triaged Won't Fix
2021-04-28 03:47:45 Billy Olsen removed subscriber Canonical Field High