Nova Cell - Nova-compute services cannot register themserves

[Paulo Duarte]

Hello Everyone,

Facing this issue while nova-compute register the services:

TASK [nova-cell : Waiting for nova-compute services to register themselves] ****************************
FAILED - RETRYING: Waiting for nova-compute services to register themselves (20 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (19 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (18 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (17 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (16 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (15 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (14 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (13 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (12 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (11 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (10 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (9 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (8 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (7 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (6 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (5 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (4 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (3 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (2 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves (1 retries left).
fatal: [10.200.201.10 -> 10.200.201.10]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["docker", "exec", "kolla_toolbox", "openstack", "--os-interface", "admin", "--os-auth-url", "http://10.200.201.10:35357", "--os-identity-api-version", "3", "--os-project-domain-name", "default", "--os-project-name", "admin", "--os-username", "admin", "--os-password", "ICvPHdiMN89ICIZ3E1HjT8R6E8L9oiD8o50ENJWX", "--os-user-domain-name", "default", "--os-region-name", "RegionOne", "compute", "service", "list", "--format", "json", "--column", "Host", "--service", "nova-compute"], "delta": "0:00:01.674758", "end": "2022-01-18 16:01:09.552255", "rc": 0, "start": "2022-01-18 16:01:07.877497", "stderr": "", "stderr_lines": [], "stdout": "[]", "stdout_lines": ["[]"]}

Looks like the nova containers cannot authenticate to keystone:

Pls find beloy the logs from nova-conductor:

2022-01-18 15:57:13.784 7 CRITICAL nova [-] Unhandled error: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d4bc3543-f2d6-47a7-b8fb-f17fbf1511a2)
2022-01-18 15:57:13.784 7 ERROR nova Traceback (most recent call last):
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2022-01-18 15:57:13.784 7 ERROR nova sys.exit(main())
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/cmd/conductor.py", line 45, in main
2022-01-18 15:57:13.784 7 ERROR nova server = service.Service.create(binary='nova-conductor',
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 252, in create
2022-01-18 15:57:13.784 7 ERROR nova service_obj = cls(host, binary, topic, manager,
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 116, in __init__
2022-01-18 15:57:13.784 7 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 121, in __init__
2022-01-18 15:57:13.784 7 ERROR nova self.compute_task_mgr = ComputeTaskManager()
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 247, in __init__
2022-01-18 15:57:13.784 7 ERROR nova self.report_client = report.SchedulerReportClient()
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 188, in __init__
2022-01-18 15:57:13.784 7 ERROR nova self._client = self._create_client()
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 231, in _create_client
2022-01-18 15:57:13.784 7 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/nova/utils.py", line 990, in get_sdk_adapter
2022-01-18 15:57:13.784 7 ERROR nova return getattr(conn, service_type)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 93, in __get__
2022-01-18 15:57:13.784 7 ERROR nova endpoint = proxy_mod.Proxy.get_endpoint(proxy)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 291, in get_endpoint
2022-01-18 15:57:13.784 7 ERROR nova return self.session.get_endpoint(auth or self.auth, **kwargs)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1233, in get_endpoint
2022-01-18 15:57:13.784 7 ERROR nova return auth.get_endpoint(self, **kwargs)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2022-01-18 15:57:13.784 7 ERROR nova endpoint_data = self.get_endpoint_data(
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2022-01-18 15:57:13.784 7 ERROR nova service_catalog = self.get_access(session).service_catalog
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2022-01-18 15:57:13.784 7 ERROR nova self.auth_ref = self.get_auth_ref(session)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2022-01-18 15:57:13.784 7 ERROR nova return self._plugin.get_auth_ref(session, **kwargs)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2022-01-18 15:57:13.784 7 ERROR nova resp = session.post(token_url, json=body, headers=headers,
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1139, in post
2022-01-18 15:57:13.784 7 ERROR nova return self.request(url, 'POST', **kwargs)
2022-01-18 15:57:13.784 7 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 976, in request
2022-01-18 15:57:13.784 7 ERROR nova raise exceptions.from_response(resp, method, url)
2022-01-18 15:57:13.784 7 ERROR nova keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d4bc3543-f2d6-47a7-b8fb-f17fbf1511a2)
2022-01-18 15:57:13.784 7 ERROR nova

Has anyone else had this problem or know how to fix it?

Thanks,

[João Pedro Seara (jpseara)]

Currently facing the same issue.

[Radosław Piliszek (yoctozepto)]

Which OpenStack release is this, which distro and whether source/binary (images)?
Do other nova parts (API) report similar problems?
What about other services?
Is this a fresh deploy?

[Gregory DeCecco (turranx)]

I am having the same problem in my lab environment.

A little bit about my lab environment:

    Kolla nodes are VMs in Hyper-V
        Beachhead
        Kolla1
        Kolla2
        Kolla3
    Beachhead Roles:
        Deployment orchestrator
        DNS Server (Bind9)
    Kolla Ansible OpenStack (Yoga 14.3.0)
    Ceph (Quincy 17.1.0)
    Ubuntu 20.04 LTS
    Python version 3.8.10
    Ansible version 2.12.8 (core) or 5.x (community)

Hosting everything in Hyper-V lets me roll back VMs to a previous state and try again, saving massive amounts of time. Plus, I can use PowerShell to script the creation of Hyper-V virtual switches, and all VMs.

This is a fresh deployment.

I dug through the logs on Kolla1 after the deployment failed and found these entries:

Kolla1: \var\lib\docker\volumes\kolla_logs\_data\libvirt\libvirtd.log
2022-08-30 20:51:59.832+0000: 58177: error : virNetSASLSessionServerStep:603 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)
2022-08-30 20:51:59.832+0000: 58177: error : remoteDispatchAuthSaslStep:3957 : authentication failed: authentication failed
2022-08-30 20:51:59.833+0000: 58156: error : virNetSocketReadWire:1793 : End of file while reading data: Input/output error

Kolla1: \var\lib\docker\volumes\kolla_logs\_data\keystone\keystone-apache-public-error.log
2022-08-30 20:47:55.800752 2022-08-30 20:47:55.800 21 WARNING keystone.server.flask.application [req-098b3bd8-1649-4f1a-b450-d34d31f41b4b 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:47:56.253671 2022-08-30 20:47:56.253 22 WARNING keystone.server.flask.application [req-bd8cb03c-1e7d-4a8a-8a81-32ef63ac3fd3 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder.\x1b[00m
2022-08-30 20:48:54.093718 2022-08-30 20:48:54.093 22 WARNING keystone.server.flask.application [req-21595d95-6230-4626-a57c-4382c89af802 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:48:58.704259 2022-08-30 20:48:58.702 21 WARNING keystone.server.flask.application [req-72852ef4-b6a1-4ecb-b4f1-b9053714939d 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:50:29.298592 2022-08-30 20:50:29.298 22 WARNING keystone.server.flask.application [req-591cab45-60ea-4e7a-bb4c-c0d1ac9a07f0 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:50:29.582448 2022-08-30 20:50:29.582 21 WARNING keystone.server.flask.application [req-38301601-3955-426a-b557-0e398ab79f29 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova.\x1b[00m

Kolla1: \var\lib\docker\volumes\kolla_logs\_data\mariadb\mariadb.log
2022-08-30 ...

[Gregory DeCecco (turranx)]

Is it possible this error is caused by not completing the few steps where they register client.cinder.key with libvirt?

https://docs.ceph.com/en/latest/rbd/rbd-openstack/#setup-ceph-client-authentication

[Launchpad Janitor (janitor)]

[Expired for kolla-ansible because there has been no activity for 60 days.]