Activity log for bug #1949891

Date Who What changed Old value New value Message
2021-11-04 18:44:57 Vahid Alimohamadi bug added bug
2021-11-05 12:54:38 Vahid Alimohamadi description What happened: When trying to create a stack using monasca and cloudkitty there's error. What you expected to happen: no errors. OS: Ubuntu 21.04 (Hirsute Hippo) Kernel: Linux os 5.11.0-38-generic #42-Ubuntu SMP Fri Sep 24 14:03:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Docker version: 20.10.9, build c2ea9bc Kolla-Ansible version: both 12.2.0 and 13.0.0.0rc1 openstack version: both wallaby and xena Docker image Install type: source globals.yml: config_strategy: "COPY_ALWAYS" kolla_base_distro: "ubuntu" kolla_install_type: "source" openstack_release: "wallaby" kolla_internal_vip_address: 192.168.150.150 network_interface: eno1 neutron_external_interface: eno2 neutron_plugin_agent: "openvswitch" enable_haproxy: "no" #enable_mariadb_clustercheck: yes #enable_ceilometer: "yes" enable_cinder: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" enable_cloudkitty: "yes" #enable_gnocchi: "yes" #enable_prometheus: "yes" enable_monasca: "yes" cloudkitty_collector_backend: monasca enable_neutron_provider_networks: "yes" ceph_cinder_keyring: "ceph.client.admin.keyring" ceph_cinder_user: "admin" ceph_cinder_pool_name: "volumes" fernet_token_expiry: 86400 cinder_backend_ceph: "yes" cinder_volume_group: "volumes" nova_compute_virt_type: "kvm" nova_console: "novnc" #enable_aodh: "yes" monasca_enable_alerting_pipeline: "no" monasca_ingest_control_plane_logs: "no" enable_horizon_monasca: "yes" kibana_log_prefix: "monasca" sudo docker logs -f cloudkitty_processor: Process Process-828: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/orchestrator.py", line 345, in __init__ self.collector = collector.get_collector() File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/collector/__init__.py", line 108, in get_collector return driver.DriverManager( File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/driver.py", line 54, in __init__ super(DriverManager, self).__init__( File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/named.py", line 78, in __init__ extensions = self._load_plugins(invoke_on_load, File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 233, in _load_plugins self._on_load_failure_callback(self, ep, err) File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 221, in _load_plugins ext = self._load_one_plugin(ep, File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/named.py", line 156, in _load_one_plugin return super(NamedExtensionManager, self)._load_one_plugin( File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 257, in _load_one_plugin obj = plugin(*invoke_args, **invoke_kwds) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/collector/monasca.py", line 90, in __init__ self._conn = mon_client_utils.get_monasca_client( File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/common/monasca_client.py", line 55, in get_monasca_client mon_endpoint = get_monasca_endpoint(conf[conf_opts], keystone_client) File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/common/monasca_client.py", line 35, in get_monasca_endpoint service_list = keystone_client.services.list(name=service_name) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/v3/services.py", line 90, in list return super(ServiceManager, self).list( File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 86, in func return f(*args, **new_kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 448, in list list_resp = self._list(url_query, self.collection_key) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 141, in _list resp, body = self.client.get(url, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 395, in get return self.request(url, 'GET', **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 554, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 257, in request return self.session.request(url, method, **kwargs) File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 811, in request base_url = self.get_endpoint(auth, allow=allow, File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1241, in get_endpoint auth = self._auth_required(auth, 'determine endpoint URL') File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1181, in _auth_required raise exceptions.MissingAuthPlugin(msg_fmt % msg) keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL I've just added metrics.yml which was missing in new images. and current configuration current cloudkitty.conf: ``` [DEFAULT] debug = True log_dir = /var/log/kolla/cloudkitty log_file = /var/log/kolla/cloudkitty/cloudkitty-api.log transport_url = rabbit://openstack:Pt4ymrgq1V114sXtixfyZYiIcaplAZhN7PUQlwK2@192.168.150.150:5672// [database] connection = mysql+pymysql://cloudkitty:ZvQofADucRK3Og0ENMAVtRfdyGiMel5iZF5LidVA@192.168.150.150:3306/cloudkitty connection_recycle_time = 10 max_pool_size = 1 max_retries = -1 [keystone_authtoken] www_authenticate_uri = http://192.168.150.150:5000 auth_url = http://192.168.150.150:35357 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = cloudkitty password = 67rfyzWSuKxqjWH2XO5oRdLezytMCAgeGMbMj46t region_name = RegionOne cafile = memcache_security_strategy = ENCRYPT memcache_secret_key = Z3iM1Jm7HrAPwoWeIO6Xhufis1DWxNGBBEkUxvFR memcached_servers = 192.168.150.150:11211 [oslo_middleware] enable_proxy_headers_parsing = True [oslo_concurrency] lock_path = /var/lib/cloudkitty/tmp [collect] collector = monasca [fetcher_keystone] keystone_version = 3 auth_section = keystone_authtoken region_name = RegionOne [collector_monasca] monasca_service_name = monasca interface = internal [api] host_ip = 192.168.150.150 port = 8889 [orchestrator] coordination_url = mysql://cloudkitty:ZvQofADucRK3Og0ENMAVtRfdyGiMel5iZF5LidVA@87.107.161.98:3306/cloudkitty max_workers = 5 [storage] backend = influxdb version = 2 [storage_influxdb] database = cloudkitty host = 192.168.150.150 port = 8086 ``` What happened: When trying to create a stack using monasca and cloudkitty there's error. What you expected to happen: no errors. OS: Ubuntu 21.04 (Hirsute Hippo) Kernel: Linux os 5.11.0-38-generic #42-Ubuntu SMP Fri Sep 24 14:03:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Docker version: 20.10.9, build c2ea9bc Kolla-Ansible version: both 12.2.0 and 13.0.0.0rc1 openstack version: both wallaby and xena Docker image Install type: source globals.yml: config_strategy: "COPY_ALWAYS" kolla_base_distro: "ubuntu" kolla_install_type: "source" openstack_release: "wallaby" kolla_internal_vip_address: 192.168.150.150 network_interface: eno1 neutron_external_interface: eno2 neutron_plugin_agent: "openvswitch" enable_haproxy: "no" #enable_mariadb_clustercheck: yes #enable_ceilometer: "yes" enable_cinder: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" enable_cloudkitty: "yes" #enable_gnocchi: "yes" #enable_prometheus: "yes" enable_monasca: "yes" cloudkitty_collector_backend: monasca enable_neutron_provider_networks: "yes" ceph_cinder_keyring: "ceph.client.admin.keyring" ceph_cinder_user: "admin" ceph_cinder_pool_name: "volumes" fernet_token_expiry: 86400 cinder_backend_ceph: "yes" cinder_volume_group: "volumes" nova_compute_virt_type: "kvm" nova_console: "novnc" #enable_aodh: "yes" monasca_enable_alerting_pipeline: "no" monasca_ingest_control_plane_logs: "no" enable_horizon_monasca: "yes" kibana_log_prefix: "monasca" sudo docker logs -f cloudkitty_processor: Process Process-828: Traceback (most recent call last):   File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap     self.run()   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process     target(*args, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_service.py", line 161, in create_and_wait     sw = cls(*args, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cotyledon/_service.py", line 175, in __init__     self.service = config.service(worker_id, *args, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/orchestrator.py", line 345, in __init__     self.collector = collector.get_collector()   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/collector/__init__.py", line 108, in get_collector     return driver.DriverManager(   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/driver.py", line 54, in __init__     super(DriverManager, self).__init__(   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/named.py", line 78, in __init__     extensions = self._load_plugins(invoke_on_load,   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 233, in _load_plugins     self._on_load_failure_callback(self, ep, err)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 221, in _load_plugins     ext = self._load_one_plugin(ep,   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/named.py", line 156, in _load_one_plugin     return super(NamedExtensionManager, self)._load_one_plugin(   File "/var/lib/kolla/venv/lib/python3.8/site-packages/stevedore/extension.py", line 257, in _load_one_plugin     obj = plugin(*invoke_args, **invoke_kwds)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/collector/monasca.py", line 90, in __init__     self._conn = mon_client_utils.get_monasca_client(   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/common/monasca_client.py", line 55, in get_monasca_client     mon_endpoint = get_monasca_endpoint(conf[conf_opts], keystone_client)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/common/monasca_client.py", line 35, in get_monasca_endpoint     service_list = keystone_client.services.list(name=service_name)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/v3/services.py", line 90, in list     return super(ServiceManager, self).list(   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 86, in func     return f(*args, **new_kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 448, in list     list_resp = self._list(url_query, self.collection_key)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneclient/base.py", line 141, in _list     resp, body = self.client.get(url, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 395, in get     return self.request(url, 'GET', **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 554, in request     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 257, in request     return self.session.request(url, method, **kwargs)   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 811, in request     base_url = self.get_endpoint(auth, allow=allow,   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1241, in get_endpoint     auth = self._auth_required(auth, 'determine endpoint URL')   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/session.py", line 1181, in _auth_required     raise exceptions.MissingAuthPlugin(msg_fmt % msg) keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL I've just added metrics.yml which was missing in new images. and current configuration current cloudkitty.conf: ``` [DEFAULT] debug = True log_dir = /var/log/kolla/cloudkitty log_file = /var/log/kolla/cloudkitty/cloudkitty-api.log transport_url = rabbit://openstack:Pt4ymrgq1V114sXtixfyZYiIcaplAZhN7PUQlwK2@192.168.150.150:5672// [database] connection = mysql+pymysql://cloudkitty:ZvQofADucRK3Og0ENMAVtRfdyGiMel5iZF5LidVA@192.168.150.150:3306/cloudkitty connection_recycle_time = 10 max_pool_size = 1 max_retries = -1 [keystone_authtoken] www_authenticate_uri = http://192.168.150.150:5000 auth_url = http://192.168.150.150:35357 auth_type = password project_domain_id = default user_domain_id = default project_name = service username = cloudkitty password = 67rfyzWSuKxqjWH2XO5oRdLezytMCAgeGMbMj46t region_name = RegionOne cafile = memcache_security_strategy = ENCRYPT memcache_secret_key = Z3iM1Jm7HrAPwoWeIO6Xhufis1DWxNGBBEkUxvFR memcached_servers = 192.168.150.150:11211 [oslo_middleware] enable_proxy_headers_parsing = True [oslo_concurrency] lock_path = /var/lib/cloudkitty/tmp [collect] collector = monasca [fetcher_keystone] keystone_version = 3 auth_section = keystone_authtoken region_name = RegionOne [collector_monasca] monasca_service_name = monasca interface = internal [api] host_ip = 192.168.150.150 port = 8889 [orchestrator] coordination_url = mysql://cloudkitty:ZvQofADucRK3Og0ENMAVtRfdyGiMel5iZF5LidVA@192.168.150.150:3306/cloudkitty max_workers = 5 [storage] backend = influxdb version = 2 [storage_influxdb] database = cloudkitty host = 192.168.150.150 port = 8086 ```