Unable to create volume with image in edge site | Glance-Cinder | Openstack DCN | Wallaby

[swogat pradhan]

Hi,
I have setup glance multistore for my infra.

glance.yaml for dcn site:

(overcloud) [stack@hkg2director ~]$ cat dcn02/glance_dcn02.yaml
parameter_defaults:
        #GlanceShowMultipleLocations: true
    GlanceEnabledImportMethods: web-download,copy-image
    GlanceBackend: rbd
    GlanceStoreDescription: 'dcn02 rbd glance store'
    GlanceBackendID: dcn02
    CephClusterName: dcn02
    GlanceMultistoreConfig:
      ceph:
        GlanceBackend: rbd
        GlanceStoreDescription: 'Default glance store backend.'
        CephClusterName: ceph

Now i have created a cirros image and have imported it to dcn store using copy-image method. When I create an empty volume in the DCN site the volume gets created without any issues.
But when I create a volume with image (volume source) the volume gets stuck in the creating state forever. I get logs in cinder-volume 3-4 mins after I have hit the create volume button.

Cinder logs:
2023-03-22 20:34:59.166 108 INFO cinder.volume.flows.manager.create_volume [req-0d2093a0-efbd-45a5-bd7d-cce25ddc200e b240e3e89d99489284cd731e75f2a5db 4160ce999a31485fa643aed0936dfef0 - - -] Volume 5743a879-090d-46db-bc7c-1c0b0669a112: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-5743a879-090d-46db-bc7c-1c0b0669a112', 'volume_size': 2, 'image_id': 'acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b', 'image_location': ('rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', [{'url': 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', 'metadata': {'store': 'ceph'}}, {'url': 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', 'metadata': {'store': 'dcn02'}}]), 'image_meta': {'name': 'cirros', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 16338944, 'virtual_size': 117440512, 'status': 'active', 'checksum': '1d3062cd89af34e419f7100277f38b2b', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '4160ce999a31485fa643aed0936dfef0', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '553d220ed58cfee7dafe003c446a9f197ab5edf8ffc09396c74187cf83873c877e7ae041cb80f3b91489acf687183adcd689b53b38e3ddd22e627e7f98a09c46', 'id': 'acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b', 'created_at': datetime.datetime(2023, 3, 22, 18, 50, 5, tzinfo=datetime.timezone.utc), 'updated_at': datetime.datetime(2023, 3, 22, 20, 3, 54, tzinfo=datetime.timezone.utc), 'locations': [{'url': 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', 'metadata': {'store': 'ceph'}}, {'url': 'rbd://a8d5f1f5-48e7-5ede-89ab-8aca59b6397b/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', 'metadata': {'store': 'dcn02'}}], 'direct_url': 'rbd://a5ae877c-bcba-53fe-8336-450e63014757/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/snap', 'tags': [], 'file': '/v2/images/acfd0a14-69e0-44d6-a6a1-aa9dc83e9d5b/file', 'stores': 'ceph,dcn02', 'properties': {'os_glance_failed_import': '', 'os_glance_importing_to_stores': '', 'owner_specified.openstack.md5': '', 'owner_specified.openstack.object': 'images/cirros', 'owner_specified.openstack.sha256': ''}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f8147973438>}

I checked both glance and cinder containers are running in a healthy state.
I see no errors or whatsoever. I am not sure how to fix the cinder volume stuck in the creating state in the DCN edge site.

[swogat pradhan (swogat)]

Cinder volume config:

[tripleo_ceph]
volume_backend_name=tripleo_ceph
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_user=openstack
rbd_pool=volumes
rbd_flatten_volume_from_snapshot=False
rbd_secret_uuid=a8d5f1f5-48e7-5ede-89ab-8aca59b6397b
report_discard_supported=True
rbd_ceph_conf=/etc/ceph/dcn02.conf
rbd_cluster_name=dcn02

Glance api config:

[dcn02]
rbd_store_ceph_conf=/etc/ceph/dcn02.conf
rbd_store_user=openstack
rbd_store_pool=images
rbd_thin_provisioning=False
store_description=dcn02 rbd glance store
[ceph]
rbd_store_ceph_conf=/etc/ceph/ceph.conf
rbd_store_user=openstack
rbd_store_pool=images
rbd_thin_provisioning=False
store_description=Default glance store backend.

[swogat pradhan (swogat)]

the volume stays in creating state forever, no timeouts no nothing.

fresh log entries attached