TrilioDataMoverTest.test_snapshot_workload fails with Permission denied: '/var/lib/charm/trilio-data-mover'

[Aurelien Lourot]

This is now happening consistently on https://review.opendev.org/#/c/748807/ . This review has a special func-test-pr https://github.com/openstack-charmers/zaza-openstack-tests/pull/400 showing us the full status of the resource in error state:

+----------------+-------------------------------------------------------------+
| Instances | Value |
+----------------+-------------------------------------------------------------+
| Name | zaza-triliovault-tests-server |
| Status | error |
| Flavor | {'vcpus': '1', 'ram': '512', 'disk': '1', 'ephemeral': '0'} |
| Security Group | [{'name': 'default', 'security_group_type': 'neutron'}] |
| ID | 10fd628e-7e22-4970-83ad-b824291879f4 |
| | |
+----------------+-------------------------------------------------------------+
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| id | bef4dc8b-3fa0-4027-a88c-568dfa304f60 |
| status | error |
| snapshot_type | full |
| workload_id | ca0283d3-fa79-471a-88e8-3e50b30ca34d |
| name | Snapshot |
| description | no-description |
| host | juju-13b05f-zaza-e598d2fbe320-21 |
| size | 150863872 Bytes or Approx (143.9MB) |
| uploaded_size | 150863872 |
| progress_percent | 100 |
| error_msg | Failed creating workload snapshot: Data transfer failed - Error: [Errno 13] Permission denied: '/var/lib/charm/trilio-data-mover' |
| time_taken | 130 Seconds |
| restores_info | |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------+

[James Page (james-page)]

ion denied: '/var/lib/charm/trilio-data-mover': PermissionError: [Errno 13] Permission denied: '/var/lib/charm/trilio-data-mover'
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver Traceback (most recent call last):
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver File "/usr/lib/python3/dist-packages/contego/nova/extension/driver/libvirtdriver.py", line 291, in vast_data_transfer
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver params=params,
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver File "/usr/lib/python3/dist-packages/contego/nova/extension/driver/backends/rdb_backend.py", line 104, in upload_snapshot
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver vsize = self._size(disk_info["path"])
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver File "/usr/lib/python3/dist-packages/contego/nova/extension/driver/backends/rbd_base.py", line 335, in _size
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver "info", *args, **kwargs
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver File "/usr/lib/python3/dist-packages/contego/nova/extension/driver/backends/rbd_base.py", line 149, in rbd_keyring_search_and_execute
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver for keyring in os.listdir(cephdir):
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver PermissionError: [Errno 13] Permission denied: '/var/lib/charm/trilio-data-mover'
2020-08-29 18:01:35.353 9332 ERROR contego.nova.extension.driver.libvirtdriver

[Aurelien Lourot (aurelien-lourot)]

Fixed by https://review.opendev.org/#/c/748807/