jammy-yoga test_cinder_volume_backup_create_delete fails assert resource_status == expected_status

[Corey Bryant]

== test_cinder_volume_backup_create_delete fails as seen in gate here ==

https://review.opendev.org/c/openstack/charm-cinder-backup-swift-proxy/+/850176

== test failure shows ==

2022-08-30 13:30:53 [INFO] ======================================================================
2022-08-30 13:30:53 [INFO] FAIL: test_cinder_volume_backup_create_delete (zaza.openstack.charm_tests.cinder_backup_swift_proxy.tests.CinderBackupSwiftTest)
2022-08-30 13:30:53 [INFO] Create volume backup and then delete it.
2022-08-30 13:30:53 [INFO] ----------------------------------------------------------------------
2022-08-30 13:30:53 [INFO] Traceback (most recent call last):
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/cinder_backup_swift_proxy/tests.py", line 45, in test_cinder_volume_backup_create_delete
2022-08-30 13:30:53 [INFO] volume_backup = openstack_utils.create_volume_backup(
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 2718, in create_volume_backup
2022-08-30 13:30:53 [INFO] resource_reaches_status(
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 2389, in resource_reaches_status
2022-08-30 13:30:53 [INFO] retryer(
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 404, in __call__
2022-08-30 13:30:53 [INFO] do = self.iter(retry_state=retry_state)
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
2022-08-30 13:30:53 [INFO] raise retry_exc.reraise()
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 193, in reraise
2022-08-30 13:30:53 [INFO] raise self.last_attempt.result()
2022-08-30 13:30:53 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
2022-08-30 13:30:53 [INFO] return self.__get_result()
2022-08-30 13:30:53 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2022-08-30 13:30:53 [INFO] raise self._exception
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 407, in __call__
2022-08-30 13:30:53 [INFO] result = fn(*args, **kwargs)
2022-08-30 13:30:53 [INFO] File "/mnt/release-tools/charms/cinder-backup-swift-proxy/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 2344, in _resource_reaches_status
2022-08-30 13:30:53 [INFO] assert resource_status == expected_status
2022-08-30 13:30:53 [INFO] AssertionError
2022-08-30 13:30:53 [INFO] ----------------------------------------------------------------------
2022-08-30 13:30:53 [INFO] Ran 1 test in 155.523s
2022-08-30 13:30:53 [INFO] FAILED

== /var/log/cinder/cinder-backup.log from cinder-backup-swift-proxy/0 ==

2022-08-30 13:27:49.322 29646 CRITICAL cinder [req-c824c2cf-7313-431e-973d-9a2a23142614 - - - - -] Unhandled error: oslo_config.cfg.ConfigFileValueError: Value for option backup_swift_auth_url from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/cinder/cinder.conf') is not valid: ('host, scheme were required but missing', URIReference(scheme=None, authority=None, path='None', query=None, fragment=None), ['host', 'scheme'])
2022-08-30 13:27:49.322 29646 ERROR cinder Traceback (most recent call last):
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/types.py", line 942, in __call__
2022-08-30 13:27:49.322 29646 ERROR cinder validator.validate(uri)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/rfc3986/validators.py", line 228, in validate
2022-08-30 13:27:49.322 29646 ERROR cinder ensure_required_components_exist(uri, required_components)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/rfc3986/validators.py", line 269, in ensure_required_components_exist
2022-08-30 13:27:49.322 29646 ERROR cinder raise exceptions.MissingComponentError(uri, *missing_components)
2022-08-30 13:27:49.322 29646 ERROR cinder rfc3986.exceptions.MissingComponentError: ('host, scheme were required but missing', URIReference(scheme=None, authority=None, path='None', query=None, fragment=None), ['host', 'scheme'])
2022-08-30 13:27:49.322 29646 ERROR cinder
2022-08-30 13:27:49.322 29646 ERROR cinder During handling of the above exception, another exception occurred:
2022-08-30 13:27:49.322 29646 ERROR cinder
2022-08-30 13:27:49.322 29646 ERROR cinder Traceback (most recent call last):
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2716, in _do_get
2022-08-30 13:27:49.322 29646 ERROR cinder return (convert(val), alt_loc)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2685, in convert
2022-08-30 13:27:49.322 29646 ERROR cinder return self._convert_value(
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2813, in _convert_value
2022-08-30 13:27:49.322 29646 ERROR cinder return opt.type(value)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/types.py", line 944, in __call__
2022-08-30 13:27:49.322 29646 ERROR cinder raise ValueError(exc)
2022-08-30 13:27:49.322 29646 ERROR cinder ValueError: ('host, scheme were required but missing', URIReference(scheme=None, authority=None, path='None', query=None, fragment=None), ['host', 'scheme'])
2022-08-30 13:27:49.322 29646 ERROR cinder
2022-08-30 13:27:49.322 29646 ERROR cinder During handling of the above exception, another exception occurred:
2022-08-30 13:27:49.322 29646 ERROR cinder
2022-08-30 13:27:49.322 29646 ERROR cinder Traceback (most recent call last):
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/bin/cinder-backup", line 10, in <module>
2022-08-30 13:27:49.322 29646 ERROR cinder sys.exit(main())
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/cinder/cmd/backup.py", line 122, in main
2022-08-30 13:27:49.322 29646 ERROR cinder launcher.wait()
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 650, in wait
2022-08-30 13:27:49.322 29646 ERROR cinder self.conf.log_opt_values(LOG, logging.DEBUG)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2603, in log_opt_values
2022-08-30 13:27:49.322 29646 ERROR cinder _sanitize(opt, getattr(self, opt_name)))
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2219, in __getattr__
2022-08-30 13:27:49.322 29646 ERROR cinder return self._get(name)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2653, in _get
2022-08-30 13:27:49.322 29646 ERROR cinder value, loc = self._do_get(name, group, namespace)
2022-08-30 13:27:49.322 29646 ERROR cinder File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2732, in _do_get
2022-08-30 13:27:49.322 29646 ERROR cinder raise ConfigFileValueError(message)
2022-08-30 13:27:49.322 29646 ERROR cinder oslo_config.cfg.ConfigFileValueError: Value for option backup_swift_auth_url from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/cinder/cinder.conf') is not valid: ('host, scheme were required but missing', URIReference(scheme=None, authority=None, path='None', query=None, fragment=None), ['host', 'scheme'])
2022-08-30 13:27:49.322 29646 ERROR cinder

== /etc/cinder/cinder.conf from cinder-backup-swift-proxy/0 ==

[DEFAULT]
backup_swift_auth_url = http://10.5.2.248:5000/v3

== IP address for ens3 in unit doesn't match what juju status shows ==

ubuntu@coreycb-bastion:/mnt/release-tools/charms/cinder-backup-swift-proxy$ juju status swift-keystone
Unit Workload Agent Machine Public address Ports Message
swift-keystone/0* active idle 19 10.5.2.248 5000/tcp Unit is ready

ubuntu@coreycb-bastion:/mnt/release-tools/charms/cinder-backup-swift-proxy$ juju ssh swift-keystone/0

ubuntu@juju-4a5d57-zaza-bd2d91443876-19:~$ ip a
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8942 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:ad:b5:b3 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.5.2.178/16 metric 100 brd 10.5.255.255 scope global dynamic ens3
       valid_lft 39837sec preferred_lft 39837sec

[Corey Bryant (corey.bryant)]

For some reason it appears juju has created 2 instances for machine 19:

$ openstack server list

| e14d4837-980f-4161-ba04-7088162eb5d4 | juju-4a5d57-zaza-bd2d91443876-19 | ACTIVE | coreycb_admin_net=10.5.2.248 | auto-sync/ubuntu-jammy-daily-amd64-server-20220810-disk1.img | m1.blue |
| d9917aa6-131a-4d6e-a49c-bfcd85b91bf5 | juju-4a5d57-zaza-bd2d91443876-19 | ACTIVE | coreycb_admin_net=10.5.2.178 | auto-sync/ubuntu-jammy-daily-amd64-server-20220810-disk1.img | m1.blue |

[Corey Bryant (corey.bryant)]

The jammy-yoga bundle can be found in the gerrit review linked at the top of the bug description.

[Corey Bryant (corey.bryant)]

This must be hitting the following juju bug: https://bugs.launchpad.net/juju/+bug/1988439