func-target: commands[0]> functest-run-suite --keep-model --bundle jammy 2024-03-15 15:19:48 [INFO] Could not determine correct model aliasfrom tests.yaml, using default 2024-03-15 15:19:52 [INFO] Deploying bundle '/home/ubuntu/src/charm-ceph-mon/tests/bundles/jammy.yaml' on to 'zaza-699f3ac4ebf9' model 2024-03-15 15:19:56 [INFO] Located charm "ceph-fs" in charm-hub, channel quincy/stable 2024-03-15 15:19:57 [INFO] Located charm "ceph-mon" in charm-hub, channel quincy/stable 2024-03-15 15:19:58 [INFO] Located charm "ceph-osd" in charm-hub, channel quincy/stable 2024-03-15 15:19:59 [INFO] Located charm "cinder" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:01 [INFO] Located charm "cinder-ceph" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:01 [INFO] Located charm "mysql-router" in charm-hub, channel 8.0/edge 2024-03-15 15:20:03 [INFO] Located charm "glance" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:04 [INFO] Located charm "mysql-router" in charm-hub, channel 8.0/edge 2024-03-15 15:20:05 [INFO] Located charm "keystone" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:05 [INFO] Located charm "mysql-router" in charm-hub, channel 8.0/edge 2024-03-15 15:20:06 [INFO] Located charm "mysql-innodb-cluster" in charm-hub, channel 8.0/edge 2024-03-15 15:20:07 [INFO] Located charm "nova-cloud-controller" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:08 [INFO] Located charm "mysql-router" in charm-hub, channel 8.0/edge 2024-03-15 15:20:09 [INFO] Located charm "nova-compute" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:10 [INFO] Located charm "placement" in charm-hub, channel 2023.1/stable 2024-03-15 15:20:11 [INFO] Located charm "mysql-router" in charm-hub, channel 8.0/edge 2024-03-15 15:20:12 [INFO] Located charm "prometheus2" in charm-hub, channel stable 2024-03-15 15:20:13 [INFO] Located charm "rabbitmq-server" in charm-hub, channel 3.9/stable 2024-03-15 15:20:33 [INFO] Executing changes: 2024-03-15 15:20:33 [INFO] - upload charm ceph-fs from charm-hub for series jammy from channel quincy/edge with architecture=amd64 2024-03-15 15:20:36 [INFO] - deploy application ceph-fs from charm-hub on jammy with quincy/edge 2024-03-15 15:20:37 [INFO] - upload charm ceph-mon from charm-hub for series jammy from channel quincy/edge with architecture=amd64 2024-03-15 15:20:39 [INFO] - deploy application ceph-mon from charm-hub on jammy with quincy/edge 2024-03-15 15:20:40 [INFO] added resource alert-rules 2024-03-15 15:20:40 [INFO] - upload charm ceph-osd from charm-hub for series jammy from channel quincy/edge with architecture=amd64 2024-03-15 15:20:42 [INFO] - deploy application ceph-osd from charm-hub on jammy with quincy/edge 2024-03-15 15:20:43 [INFO] - upload charm cinder from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:20:45 [INFO] - deploy application cinder from charm-hub on jammy with 2023.1/edge 2024-03-15 15:20:46 [INFO] added resource policyd-override 2024-03-15 15:20:46 [INFO] - expose all endpoints of cinder and allow access from CIDRs 0.0.0.0/0 and ::/0 2024-03-15 15:20:46 [INFO] - upload charm cinder-ceph from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:20:47 [INFO] - deploy application cinder-ceph from charm-hub on jammy with 2023.1/edge 2024-03-15 15:20:48 [INFO] - upload charm mysql-router from charm-hub for series jammy from channel 8.0/edge with architecture=amd64 2024-03-15 15:20:49 [INFO] - deploy application cinder-mysql-router from charm-hub on jammy with 8.0/edge using mysql-router 2024-03-15 15:20:50 [INFO] - upload charm glance from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:20:52 [INFO] - deploy application glance from charm-hub on jammy with 2023.1/edge 2024-03-15 15:20:52 [INFO] added resource policyd-override 2024-03-15 15:20:52 [INFO] - expose all endpoints of glance and allow access from CIDRs 0.0.0.0/0 and ::/0 2024-03-15 15:20:53 [INFO] - deploy application glance-mysql-router from charm-hub on jammy with 8.0/edge using mysql-router 2024-03-15 15:20:53 [INFO] - upload charm keystone from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:20:55 [INFO] - deploy application keystone from charm-hub on jammy with 2023.1/edge 2024-03-15 15:20:56 [INFO] added resource policyd-override 2024-03-15 15:20:56 [INFO] - expose all endpoints of keystone and allow access from CIDRs 0.0.0.0/0 and ::/0 2024-03-15 15:20:56 [INFO] - deploy application keystone-mysql-router from charm-hub on jammy with 8.0/edge using mysql-router 2024-03-15 15:20:56 [INFO] - upload charm mysql-innodb-cluster from charm-hub for series jammy from channel 8.0/edge with architecture=amd64 2024-03-15 15:20:59 [INFO] - deploy application mysql-innodb-cluster from charm-hub on jammy with 8.0/edge 2024-03-15 15:20:59 [INFO] added resource mysql-shell 2024-03-15 15:20:59 [INFO] added resource mysqld-exporter 2024-03-15 15:20:59 [INFO] - upload charm nova-cloud-controller from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:21:01 [INFO] - deploy application nova-cloud-controller from charm-hub on jammy with 2023.1/edge 2024-03-15 15:21:02 [INFO] added resource policyd-override 2024-03-15 15:21:02 [INFO] - expose all endpoints of nova-cloud-controller and allow access from CIDRs 0.0.0.0/0 and ::/0 2024-03-15 15:21:02 [INFO] - deploy application nova-cloud-controller-mysql-router from charm-hub on jammy with 8.0/edge using mysql-router 2024-03-15 15:21:03 [INFO] - upload charm nova-compute from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:21:04 [INFO] - deploy application nova-compute from charm-hub on jammy with 2023.1/edge 2024-03-15 15:21:05 [INFO] - upload charm placement from charm-hub for series jammy from channel 2023.1/edge with architecture=amd64 2024-03-15 15:21:07 [INFO] - deploy application placement from charm-hub on jammy with 2023.1/edge 2024-03-15 15:21:07 [INFO] - deploy application placement-mysql-router from charm-hub on jammy with 8.0/edge using mysql-router 2024-03-15 15:21:08 [INFO] - upload charm prometheus2 from charm-hub for series jammy with architecture=amd64 2024-03-15 15:21:09 [INFO] - deploy application prometheus2 from charm-hub on jammy 2024-03-15 15:21:10 [INFO] added resource core 2024-03-15 15:21:10 [INFO] added resource prometheus 2024-03-15 15:21:10 [INFO] - upload charm rabbitmq-server from charm-hub for series jammy from channel 3.9/edge with architecture=amd64 2024-03-15 15:21:12 [INFO] - deploy application rabbitmq-server from charm-hub on jammy with 3.9/edge 2024-03-15 15:21:13 [INFO] - add new machine 0 2024-03-15 15:21:13 [INFO] - add new machine 1 2024-03-15 15:21:13 [INFO] - add new machine 2 2024-03-15 15:21:13 [INFO] - add new machine 3 2024-03-15 15:21:13 [INFO] - add new machine 4 2024-03-15 15:21:13 [INFO] - add new machine 5 2024-03-15 15:21:13 [INFO] - add new machine 6 2024-03-15 15:21:13 [INFO] - add new machine 7 2024-03-15 15:21:14 [INFO] - add new machine 8 2024-03-15 15:21:14 [INFO] - add new machine 9 2024-03-15 15:21:14 [INFO] - add new machine 10 2024-03-15 15:21:14 [INFO] - add new machine 11 2024-03-15 15:21:14 [INFO] - add new machine 12 2024-03-15 15:21:14 [INFO] - add new machine 13 2024-03-15 15:21:14 [INFO] - add new machine 14 2024-03-15 15:21:15 [INFO] - add new machine 15 2024-03-15 15:21:15 [INFO] - add new machine 16 2024-03-15 15:21:15 [INFO] - add new machine 17 2024-03-15 15:21:15 [INFO] - add relation nova-compute:amqp - rabbitmq-server:amqp 2024-03-15 15:21:15 [INFO] - add relation nova-compute:image-service - glance:image-service 2024-03-15 15:21:15 [INFO] - add relation nova-compute:ceph - ceph-mon:client 2024-03-15 15:21:15 [INFO] - add relation nova-compute:ceph-access - cinder-ceph:ceph-access 2024-03-15 15:21:15 [INFO] - add relation keystone:shared-db - keystone-mysql-router:shared-db 2024-03-15 15:21:16 [INFO] - add relation keystone-mysql-router:db-router - mysql-innodb-cluster:db-router 2024-03-15 15:21:16 [INFO] - add relation glance:shared-db - glance-mysql-router:shared-db 2024-03-15 15:21:16 [INFO] - add relation glance-mysql-router:db-router - mysql-innodb-cluster:db-router 2024-03-15 15:21:16 [INFO] - add relation glance:identity-service - keystone:identity-service 2024-03-15 15:21:16 [INFO] - add relation glance:amqp - rabbitmq-server:amqp 2024-03-15 15:21:16 [INFO] - add relation glance:ceph - ceph-mon:client 2024-03-15 15:21:16 [INFO] - add relation cinder:shared-db - cinder-mysql-router:shared-db 2024-03-15 15:21:16 [INFO] - add relation cinder-mysql-router:db-router - mysql-innodb-cluster:db-router 2024-03-15 15:21:17 [INFO] - add relation cinder:identity-service - keystone:identity-service 2024-03-15 15:21:17 [INFO] - add relation cinder:amqp - rabbitmq-server:amqp 2024-03-15 15:21:17 [INFO] - add relation cinder:image-service - glance:image-service 2024-03-15 15:21:17 [INFO] - add relation cinder-ceph:storage-backend - cinder:storage-backend 2024-03-15 15:21:17 [INFO] - add relation cinder-ceph:ceph - ceph-mon:client 2024-03-15 15:21:17 [INFO] - add relation ceph-osd:mon - ceph-mon:osd 2024-03-15 15:21:17 [INFO] - add relation ceph-mon:mds - ceph-fs:ceph-mds 2024-03-15 15:21:17 [INFO] - add relation nova-cloud-controller:shared-db - nova-cloud-controller-mysql-router:shared-db 2024-03-15 15:21:18 [INFO] - add relation nova-cloud-controller-mysql-router:db-router - mysql-innodb-cluster:db-router 2024-03-15 15:21:18 [INFO] - add relation nova-cloud-controller:identity-service - keystone:identity-service 2024-03-15 15:21:18 [INFO] - add relation nova-cloud-controller:amqp - rabbitmq-server:amqp 2024-03-15 15:21:18 [INFO] - add relation nova-cloud-controller:cloud-compute - nova-compute:cloud-compute 2024-03-15 15:21:18 [INFO] - add relation nova-cloud-controller:image-service - glance:image-service 2024-03-15 15:21:18 [INFO] - add relation placement:shared-db - placement-mysql-router:shared-db 2024-03-15 15:21:18 [INFO] - add relation placement-mysql-router:db-router - mysql-innodb-cluster:db-router 2024-03-15 15:21:18 [INFO] - add relation placement - keystone 2024-03-15 15:21:19 [INFO] - add relation placement - nova-cloud-controller 2024-03-15 15:21:19 [INFO] - add relation ceph-mon:prometheus - prometheus2:target 2024-03-15 15:21:19 [INFO] - add unit ceph-fs/0 to new machine 17 2024-03-15 15:21:19 [INFO] - add unit ceph-mon/0 to new machine 6 2024-03-15 15:21:19 [INFO] - add unit ceph-mon/1 to new machine 7 2024-03-15 15:21:20 [INFO] - add unit ceph-mon/2 to new machine 8 2024-03-15 15:21:20 [INFO] - add unit ceph-osd/0 to new machine 3 2024-03-15 15:21:21 [INFO] - add unit ceph-osd/1 to new machine 4 2024-03-15 15:21:22 [INFO] - add unit ceph-osd/2 to new machine 5 2024-03-15 15:21:23 [INFO] - add unit cinder/0 to new machine 13 2024-03-15 15:21:23 [INFO] - add unit glance/0 to new machine 12 2024-03-15 15:21:24 [INFO] - add unit keystone/0 to new machine 10 2024-03-15 15:21:24 [INFO] - add unit mysql-innodb-cluster/0 to new machine 0 2024-03-15 15:21:24 [INFO] - add unit mysql-innodb-cluster/1 to new machine 1 2024-03-15 15:21:24 [INFO] - add unit mysql-innodb-cluster/2 to new machine 2 2024-03-15 15:21:24 [INFO] - add unit nova-cloud-controller/0 to new machine 14 2024-03-15 15:21:25 [INFO] - add unit nova-compute/0 to new machine 11 2024-03-15 15:21:25 [INFO] - add unit placement/0 to new machine 15 2024-03-15 15:21:25 [INFO] - add unit prometheus2/0 to new machine 16 2024-03-15 15:21:25 [INFO] - add unit rabbitmq-server/0 to new machine 9 2024-03-15 15:21:25 [INFO] Deploy of bundle completed. 2024-03-15 15:21:26 [INFO] Waiting for environment to settle 2024-03-15 15:21:26 [INFO] Timeout for deployment to settle set to: 3600 2024-03-15 15:21:26 [INFO] Maximum resolve count for deployment set to: 0 2024-03-15 15:21:26 [INFO] Waiting for application states to reach targeted states. 2024-03-15 15:21:26 [INFO] Waiting for an application to be present 2024-03-15 15:21:26 [INFO] Now checking workload status and status messages 2024-03-15 15:25:56 [INFO] Application rabbitmq-server is ready. 2024-03-15 15:27:01 [INFO] Application mysql-innodb-cluster is ready. 2024-03-15 15:28:02 [INFO] Application ceph-osd is ready. 2024-03-15 15:29:00 [INFO] Application ceph-mon is ready. 2024-03-15 15:30:50 [INFO] Application ceph-fs is ready. 2024-03-15 15:31:27 [INFO] Applications left: prometheus2, placement, keystone-mysql-router, glance, cinder-ceph, glance-mysql-router, nova-compute, keystone, nova-cloud-controller-mysql-router, placement-mysql-router, nova-cloud-controller, cinder-mysql-router, cinder 2024-03-15 15:33:36 [INFO] Application prometheus2 is ready. 2024-03-15 15:35:34 [INFO] Application placement-mysql-router is ready. 2024-03-15 15:36:39 [INFO] Application nova-compute is ready. 2024-03-15 15:39:42 [INFO] Application glance-mysql-router is ready. 2024-03-15 15:41:28 [INFO] Applications left: placement, keystone-mysql-router, glance, cinder-ceph, keystone, nova-cloud-controller-mysql-router, nova-cloud-controller, cinder-mysql-router, cinder 2024-03-15 15:42:50 [INFO] Application nova-cloud-controller-mysql-router is ready. 2024-03-15 15:46:14 [INFO] Application placement is ready. 2024-03-15 15:46:19 [INFO] Application keystone is ready. 2024-03-15 15:46:22 [INFO] Application keystone-mysql-router is ready. 2024-03-15 15:46:27 [INFO] Application glance is ready. 2024-03-15 15:51:05 [INFO] Application nova-cloud-controller is ready. 2024-03-15 15:51:28 [INFO] Applications left: cinder-ceph, cinder-mysql-router, cinder 2024-03-15 16:01:28 [INFO] Applications left: cinder-ceph, cinder-mysql-router, cinder 2024-03-15 16:02:03 [INFO] Application cinder is ready. 2024-03-15 16:02:20 [INFO] Application cinder-mysql-router is ready. 2024-03-15 16:02:21 [INFO] Application cinder-ceph is ready. 2024-03-15 16:02:21 [INFO] All applications reached approved status, number of units (where relevant), and workload status message checks. 2024-03-15 16:02:21 [INFO] Waiting for zaza-699f3ac4ebf9 to settle 2024-03-15 16:02:57 [INFO] Model zaza-699f3ac4ebf9 has settled 2024-03-15 16:02:57 [INFO] configuring zaza-699f3ac4ebf9 2024-03-15 16:02:57 [INFO] Finished configuring zaza-699f3ac4ebf9 2024-03-15 16:02:57 [INFO] Testing zaza-699f3ac4ebf9 2024-03-15 16:02:57 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes ## 2024-03-15 16:02:58 [INFO] test_check_pool_types (zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes) 2024-03-15 16:02:58 [INFO] Check type of pools created for clients. 2024-03-15 16:02:58 [INFO] ... 2024-03-15 16:03:00 [INFO] Checking glance is replicated 2024-03-15 16:03:00 [INFO] Checking nova is replicated 2024-03-15 16:03:00 [INFO] Checking cinder-ceph is replicated 2024-03-15 16:03:00 [INFO] ok 2024-03-15 16:03:00 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:03:00 [INFO] Ran 1 test in 1.535s 2024-03-15 16:03:00 [INFO] OK 2024-03-15 16:03:00 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest ## 2024-03-15 16:03:02 [INFO] test_pg_tuning (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:03:02 [INFO] ... 2024-03-15 16:03:04 [INFO] ok 2024-03-15 16:03:04 [INFO] test_processes (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:03:04 [INFO] Verify Ceph processes. 2024-03-15 16:03:04 [INFO] ... 2024-03-15 16:03:04 [INFO] Checking ceph-mon and ceph-osd processes... 2024-03-15 16:03:07 [INFO] ok 2024-03-15 16:03:07 [INFO] test_services (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:03:07 [INFO] Verify the ceph services. 2024-03-15 16:03:07 [INFO] ... 2024-03-15 16:03:07 [INFO] Checking ceph-osd and ceph-mon services... 2024-03-15 16:03:09 [INFO] ok 2024-03-15 16:03:09 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:03:09 [INFO] Ran 3 tests in 9.079s 2024-03-15 16:03:09 [INFO] OK 2024-03-15 16:03:09 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephTest ## 2024-03-15 16:03:11 [INFO] test_blacklist (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:03:11 [INFO] Check the blacklist action. 2024-03-15 16:03:11 [INFO] ... 2024-03-15 16:03:11 [INFO] Checking blacklist-add-disk and blacklist-remove-disk actions... 2024-03-15 16:03:27 [INFO] ok 2024-03-15 16:03:27 [INFO] test_blocked_when_non_pristine_disk_appears (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:03:27 [INFO] Test blocked state with non-pristine disk. 2024-03-15 16:03:27 [INFO] ... 2024-03-15 16:03:39 [INFO] looking at application: {'name': 'ceph-mon', 'type': {'pkg': 'ceph-common', 'origin_setting': 'source'}} 2024-03-15 16:03:45 [WARNING] Skipping pristine disk test for focal and higher 2024-03-15 16:03:45 [INFO] ok 2024-03-15 16:03:45 [INFO] test_cache_device (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:03:45 [INFO] Test replacing a disk in use. 2024-03-15 16:03:45 [INFO] ... 2024-03-15 16:03:45 [INFO] Running add-disk action with a caching device 2024-03-15 16:03:45 [INFO] Removing single disk from each OSD 2024-03-15 16:03:48 [INFO] skipped 'Skipping OSD replacement Test, no spare devices added' 2024-03-15 16:03:48 [INFO] test_ceph_check_osd_pools (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:03:48 [INFO] Check OSD pools. 2024-03-15 16:03:48 [INFO] ... 2024-03-15 16:03:48 [INFO] Checking pools on ceph units... 2024-03-15 16:03:56 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}} 2024-03-15 16:03:58 [INFO] ok 2024-03-15 16:03:58 [INFO] test_ceph_encryption (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:03:58 [INFO] Test Ceph encryption. 2024-03-15 16:03:58 [INFO] ... 2024-03-15 16:04:08 [INFO] looking at application: {'name': 'ceph-mon', 'type': {'pkg': 'ceph-common', 'origin_setting': 'source'}} 2024-03-15 16:04:12 [WARNING] Skipping encryption test for Mitaka and higher 2024-03-15 16:04:12 [INFO] ok 2024-03-15 16:04:12 [INFO] test_ceph_pool_creation_with_text_file (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:04:12 [INFO] Check the creation of a pool and a text file. 2024-03-15 16:04:12 [INFO] ... 2024-03-15 16:04:16 [INFO] ok 2024-03-15 16:04:16 [INFO] test_list_disks (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:04:16 [INFO] Test the list-disks action. 2024-03-15 16:04:16 [INFO] ... 2024-03-15 16:04:16 [INFO] Checking list-disks action... 2024-03-15 16:04:20 [INFO] ok 2024-03-15 16:04:20 [INFO] test_pause_and_resume (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:04:20 [INFO] The services can be paused and resumed. 2024-03-15 16:04:20 [INFO] ... 2024-03-15 16:04:20 [INFO] Checking pause and resume actions... 2024-03-15 16:04:20 [INFO] ok 2024-03-15 16:04:20 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:04:20 [INFO] Ran 8 tests in 71.133s 2024-03-15 16:04:20 [INFO] OK 2024-03-15 16:04:20 [INFO] (skipped=1) 2024-03-15 16:04:20 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest ## 2024-03-15 16:04:20 [INFO] test_osd_security_checklist (zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest) 2024-03-15 16:04:20 [INFO] Verify expected state with security-checklist. 2024-03-15 16:04:20 [INFO] ... 2024-03-15 16:04:20 [INFO] Running `security-checklist` action on Ceph OSD leader unit 2024-03-15 16:04:22 [INFO] ok 2024-03-15 16:04:22 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:04:22 [INFO] Ran 1 test in 1.498s 2024-03-15 16:04:22 [INFO] OK 2024-03-15 16:04:22 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest ## 2024-03-15 16:04:22 [INFO] test_prometheus_metrics (zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest) 2024-03-15 16:04:22 [INFO] Validate that Prometheus has Ceph metrics. 2024-03-15 16:04:22 [INFO] ... 2024-03-15 16:04:23 [INFO] ok 2024-03-15 16:04:23 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:04:23 [INFO] Ran 1 test in 1.254s 2024-03-15 16:04:23 [INFO] OK 2024-03-15 16:04:23 [INFO] ## Running Test zaza.charm_tests.lifecycle.tests.UpgradeCharmsToPath;ceph-mon ## 2024-03-15 16:04:23 [INFO] Performing a charm upgrade on: ['ceph-mon'] Added local charm "ceph-mon", revision 0, to the model Leaving endpoints in "alpha": admin, bootstrap-source, client, cluster, dashboard, mds, metrics-endpoint, mon, nrpe-external-master, osd, prometheus, public, radosgw, rbd-mirror 2024-03-15 16:04:28 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephMonJujuPersistent ## 2024-03-15 16:04:30 [INFO] test_persistent_config (zaza.openstack.charm_tests.ceph.tests.CephMonJujuPersistent) 2024-03-15 16:04:30 [INFO] Check persistent config will update if config change. 2024-03-15 16:04:30 [INFO] ... 2024-03-15 16:04:30 [INFO] Waiting for at least one unit with agent status "executing" 2024-03-15 16:04:32 [INFO] Waiting for application states to reach targeted states. 2024-03-15 16:04:32 [INFO] Waiting for an application to be present 2024-03-15 16:04:32 [INFO] Now checking workload status and status messages 2024-03-15 16:04:32 [INFO] Application prometheus2 is ready. 2024-03-15 16:04:32 [INFO] Application glance is ready. 2024-03-15 16:04:33 [INFO] Application ceph-osd is ready. 2024-03-15 16:04:33 [INFO] Application ceph-fs is ready. 2024-03-15 16:04:33 [INFO] Application nova-compute is ready. 2024-03-15 16:04:33 [INFO] Application rabbitmq-server is ready. 2024-03-15 16:04:33 [INFO] Application keystone is ready. 2024-03-15 16:04:33 [INFO] Application nova-cloud-controller-mysql-router is ready. 2024-03-15 16:04:33 [INFO] Application cinder is ready. 2024-03-15 16:04:33 [INFO] Application mysql-innodb-cluster is ready. 2024-03-15 16:04:33 [INFO] Application nova-cloud-controller is ready. 2024-03-15 16:04:33 [INFO] Application placement is ready. 2024-03-15 16:04:33 [INFO] Application keystone-mysql-router is ready. 2024-03-15 16:04:33 [INFO] Application cinder-ceph is ready. 2024-03-15 16:04:33 [INFO] Application placement-mysql-router is ready. 2024-03-15 16:04:33 [INFO] Application cinder-mysql-router is ready. 2024-03-15 16:04:33 [INFO] Application glance-mysql-router is ready. 2024-03-15 16:05:45 [INFO] Application ceph-mon is ready. 2024-03-15 16:05:45 [INFO] All applications reached approved status, number of units (where relevant), and workload status message checks. 2024-03-15 16:05:46 [INFO] Waiting for at least one unit with agent status "executing" 2024-03-15 16:05:46 [INFO] Waiting for application states to reach targeted states. 2024-03-15 16:05:46 [INFO] Waiting for an application to be present 2024-03-15 16:05:46 [INFO] Now checking workload status and status messages 2024-03-15 16:05:46 [INFO] Application prometheus2 is ready. 2024-03-15 16:05:46 [INFO] Application glance is ready. 2024-03-15 16:05:47 [INFO] Application ceph-osd is ready. 2024-03-15 16:05:47 [INFO] Application ceph-fs is ready. 2024-03-15 16:05:47 [INFO] Application nova-compute is ready. 2024-03-15 16:05:47 [INFO] Application rabbitmq-server is ready. 2024-03-15 16:05:47 [INFO] Application keystone is ready. 2024-03-15 16:05:47 [INFO] Application nova-cloud-controller-mysql-router is ready. 2024-03-15 16:05:47 [INFO] Application cinder is ready. 2024-03-15 16:05:47 [INFO] Application mysql-innodb-cluster is ready. 2024-03-15 16:05:47 [INFO] Application nova-cloud-controller is ready. 2024-03-15 16:05:47 [INFO] Application placement is ready. 2024-03-15 16:05:47 [INFO] Application keystone-mysql-router is ready. 2024-03-15 16:05:47 [INFO] Application cinder-ceph is ready. 2024-03-15 16:05:47 [INFO] Application placement-mysql-router is ready. 2024-03-15 16:05:47 [INFO] Application cinder-mysql-router is ready. 2024-03-15 16:05:47 [INFO] Application glance-mysql-router is ready. 2024-03-15 16:06:02 [INFO] Application ceph-mon is ready. 2024-03-15 16:06:02 [INFO] All applications reached approved status, number of units (where relevant), and workload status message checks. 2024-03-15 16:06:02 [INFO] ok 2024-03-15 16:06:02 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:06:02 [INFO] Ran 1 test in 94.363s 2024-03-15 16:06:02 [INFO] OK 2024-03-15 16:06:02 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes ## 2024-03-15 16:06:02 [INFO] test_check_pool_types (zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes) 2024-03-15 16:06:02 [INFO] Check type of pools created for clients. 2024-03-15 16:06:02 [INFO] ... 2024-03-15 16:06:04 [INFO] Checking glance is replicated 2024-03-15 16:06:04 [INFO] Checking nova is replicated 2024-03-15 16:06:04 [INFO] Checking cinder-ceph is replicated 2024-03-15 16:06:04 [INFO] ok 2024-03-15 16:06:04 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:06:04 [INFO] Ran 1 test in 1.961s 2024-03-15 16:06:04 [INFO] OK 2024-03-15 16:06:04 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest ## 2024-03-15 16:06:06 [INFO] test_pg_tuning (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:06:06 [INFO] ... 2024-03-15 16:06:09 [INFO] ok 2024-03-15 16:06:09 [INFO] test_processes (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:06:09 [INFO] Verify Ceph processes. 2024-03-15 16:06:09 [INFO] ... 2024-03-15 16:06:09 [INFO] Checking ceph-mon and ceph-osd processes... 2024-03-15 16:06:11 [INFO] ok 2024-03-15 16:06:11 [INFO] test_services (zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest) 2024-03-15 16:06:11 [INFO] Verify the ceph services. 2024-03-15 16:06:11 [INFO] ... 2024-03-15 16:06:11 [INFO] Checking ceph-osd and ceph-mon services... 2024-03-15 16:06:13 [INFO] ok 2024-03-15 16:06:13 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:06:13 [INFO] Ran 3 tests in 9.000s 2024-03-15 16:06:13 [INFO] OK 2024-03-15 16:06:13 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephTest ## 2024-03-15 16:06:15 [INFO] test_blacklist (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:06:15 [INFO] Check the blacklist action. 2024-03-15 16:06:15 [INFO] ... 2024-03-15 16:06:15 [INFO] Checking blacklist-add-disk and blacklist-remove-disk actions... 2024-03-15 16:06:31 [INFO] ok 2024-03-15 16:06:31 [INFO] test_blocked_when_non_pristine_disk_appears (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:06:31 [INFO] Test blocked state with non-pristine disk. 2024-03-15 16:06:31 [INFO] ... 2024-03-15 16:06:44 [INFO] looking at application: {'name': 'ceph-mon', 'type': {'pkg': 'ceph-common', 'origin_setting': 'source'}} 2024-03-15 16:06:48 [WARNING] Skipping pristine disk test for focal and higher 2024-03-15 16:06:48 [INFO] ok 2024-03-15 16:06:48 [INFO] test_cache_device (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:06:48 [INFO] Test replacing a disk in use. 2024-03-15 16:06:48 [INFO] ... 2024-03-15 16:06:48 [INFO] Running add-disk action with a caching device 2024-03-15 16:06:48 [INFO] Removing single disk from each OSD 2024-03-15 16:06:51 [INFO] skipped 'Skipping OSD replacement Test, no spare devices added' 2024-03-15 16:06:51 [INFO] test_ceph_check_osd_pools (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:06:51 [INFO] Check OSD pools. 2024-03-15 16:06:51 [INFO] ... 2024-03-15 16:06:51 [INFO] Checking pools on ceph units... 2024-03-15 16:06:59 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}} 2024-03-15 16:07:01 [INFO] ok 2024-03-15 16:07:01 [INFO] test_ceph_encryption (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:07:01 [INFO] Test Ceph encryption. 2024-03-15 16:07:01 [INFO] ... 2024-03-15 16:07:11 [INFO] looking at application: {'name': 'ceph-mon', 'type': {'pkg': 'ceph-common', 'origin_setting': 'source'}} 2024-03-15 16:07:14 [WARNING] Skipping encryption test for Mitaka and higher 2024-03-15 16:07:14 [INFO] ok 2024-03-15 16:07:14 [INFO] test_ceph_pool_creation_with_text_file (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:07:14 [INFO] Check the creation of a pool and a text file. 2024-03-15 16:07:14 [INFO] ... 2024-03-15 16:07:15 [INFO] ok 2024-03-15 16:07:15 [INFO] test_list_disks (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:07:15 [INFO] Test the list-disks action. 2024-03-15 16:07:15 [INFO] ... 2024-03-15 16:07:15 [INFO] Checking list-disks action... 2024-03-15 16:07:19 [INFO] ok 2024-03-15 16:07:19 [INFO] test_pause_and_resume (zaza.openstack.charm_tests.ceph.tests.CephTest) 2024-03-15 16:07:19 [INFO] The services can be paused and resumed. 2024-03-15 16:07:19 [INFO] ... 2024-03-15 16:07:19 [INFO] Checking pause and resume actions... 2024-03-15 16:07:19 [INFO] ok 2024-03-15 16:07:19 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:07:19 [INFO] Ran 8 tests in 65.708s 2024-03-15 16:07:19 [INFO] OK 2024-03-15 16:07:19 [INFO] (skipped=1) 2024-03-15 16:07:19 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest ## 2024-03-15 16:07:19 [INFO] test_osd_security_checklist (zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest) 2024-03-15 16:07:19 [INFO] Verify expected state with security-checklist. 2024-03-15 16:07:19 [INFO] ... 2024-03-15 16:07:19 [INFO] Running `security-checklist` action on Ceph OSD leader unit 2024-03-15 16:07:21 [INFO] ok 2024-03-15 16:07:21 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:07:21 [INFO] Ran 1 test in 1.812s 2024-03-15 16:07:21 [INFO] OK 2024-03-15 16:07:21 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest ## 2024-03-15 16:07:21 [INFO] test_prometheus_metrics (zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest) 2024-03-15 16:07:21 [INFO] Validate that Prometheus has Ceph metrics. 2024-03-15 16:07:21 [INFO] ... 2024-03-15 16:07:22 [INFO] ok 2024-03-15 16:07:22 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:07:22 [INFO] Ran 1 test in 1.692s 2024-03-15 16:07:22 [INFO] OK 2024-03-15 16:07:22 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephAuthTest ## 2024-03-15 16:07:22 [INFO] test_ceph_auth (zaza.openstack.charm_tests.ceph.tests.CephAuthTest) 2024-03-15 16:07:22 [INFO] Test creating and deleting user. 2024-03-15 16:07:22 [INFO] ... 2024-03-15 16:07:22 [INFO] Creating user and exported keyring... 2024-03-15 16:07:27 [INFO] Getting existing user and exported keyring... 2024-03-15 16:07:30 [INFO] Deleting existing user... 2024-03-15 16:07:34 [INFO] Verify user is deleted... 2024-03-15 16:07:35 [INFO] ok 2024-03-15 16:07:35 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:07:35 [INFO] Ran 1 test in 12.998s 2024-03-15 16:07:35 [INFO] OK 2024-03-15 16:07:35 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephMonActionsTest ## 2024-03-15 16:07:39 [INFO] test_copy_pool (zaza.openstack.charm_tests.ceph.tests.CephMonActionsTest) 2024-03-15 16:07:39 [INFO] Test the copy-pool (and list-pool) action. 2024-03-15 16:07:39 [INFO] ... 2024-03-15 16:07:52 [INFO] ok 2024-03-15 16:07:52 [INFO] test_reweight_osd (zaza.openstack.charm_tests.ceph.tests.CephMonActionsTest) 2024-03-15 16:07:52 [INFO] Test the change-osd-weight action. 2024-03-15 16:07:52 [INFO] ... 2024-03-15 16:08:02 [INFO] ok 2024-03-15 16:08:02 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:08:02 [INFO] Ran 2 tests in 26.427s 2024-03-15 16:08:02 [INFO] OK 2024-03-15 16:08:02 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.mon.tests.CephPermissionUpgradeTest ## 2024-03-15 16:08:06 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}} 2024-03-15 16:08:06 [INFO] Using keystone API V3 (or later) for overcloud auth scp: /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt: No such file or directory 2024-03-15 16:08:08 [INFO] test_ceph_permission_upgrade (zaza.openstack.charm_tests.ceph.mon.tests.CephPermissionUpgradeTest) 2024-03-15 16:08:08 [INFO] Check that the charm updates permissions on charm upgrade. 2024-03-15 16:08:08 [INFO] ... 2024-03-15 16:08:53 [INFO] ok 2024-03-15 16:08:53 [INFO] ---------------------------------------------------------------------- 2024-03-15 16:08:53 [INFO] Ran 1 test in 51.168s 2024-03-15 16:08:53 [INFO] OK 2024-03-15 16:08:53 [INFO] Finished testing zaza-699f3ac4ebf9 2024-03-15 16:08:53 [INFO] Events: Deploy Bundle: Start: 1710515992.8099723 Finish: 1710516086.5489063 Elapsed Time: 93.73893404006958 PCT Of Run Time: 4 Prepare Environment: Start: 1710515988.2187657 Finish: 1710515992.8092358 Elapsed Time: 4.5904700756073 PCT Of Run Time: 1 Test zaza.charm_tests.lifecycle.tests.UpgradeCharmsToPath;ceph-mon: Start: 1710518663.496419 Finish: 1710518668.3889372 Elapsed Time: 4.8925182819366455 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.mon.tests.CephPermissionUpgradeTest: Start: 1710518882.371464 Finish: 1710518933.5469208 Elapsed Time: 51.17545676231384 PCT Of Run Time: 2 Test zaza.openstack.charm_tests.ceph.osd.tests.SecurityTest: Start: 1710518839.4351707 Finish: 1710518841.249194 Elapsed Time: 1.8140232563018799 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.tests.CephAuthTest: Start: 1710518842.9432013 Finish: 1710518855.94271 Elapsed Time: 12.999508619308472 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.tests.CephLowLevelTest: Start: 1710518764.7217534 Finish: 1710518773.7229624 Elapsed Time: 9.001209020614624 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.tests.CephMonActionsTest: Start: 1710518855.9428604 Finish: 1710518882.3712504 Elapsed Time: 26.42839002609253 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.tests.CephMonJujuPersistent: Start: 1710518668.3891659 Finish: 1710518762.7588964 Elapsed Time: 94.3697304725647 PCT Of Run Time: 4 Test zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest: Start: 1710518841.2492943 Finish: 1710518842.943128 Elapsed Time: 1.693833827972412 PCT Of Run Time: 1 Test zaza.openstack.charm_tests.ceph.tests.CephTest: Start: 1710518773.7234535 Finish: 1710518839.4349415 Elapsed Time: 65.71148800849915 PCT Of Run Time: 3 Test zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes: Start: 1710518762.7592592 Finish: 1710518764.7217026 Elapsed Time: 1.9624433517456055 PCT Of Run Time: 1 Wait for Deployment: Start: 1710516086.548988 Finish: 1710518541.8577774 Elapsed Time: 2455.308789253235 PCT Of Run Time: 84 Metadata: {} func-target: OK (2949.74=setup[0.30]+cmd[2949.44] seconds) congratulations :) (2950.12 seconds)