nova-api-os-compute breaks after running os-nova-install

Bug #2023370 reported by Danila Balagansky
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Unassigned

Bug Description

Running `os-nova-install` breaks `nova-api-os-compute` by updating password on every run with this task: https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/service_setup/tasks/main.yml#L80.

Steps to reproduce:
* openstack-ansible os-nova-install.yml --tags common-service
* when play reaches around `Add service users`, start creating server: `openstack server delete test1; openstack server create --network public --flavor tempest1 --image cirros test1 --wait `

Sometimes it would take a couple of tries to break (seems to be related to the amount of requests Nova serves and particular timing of `openstack.cloud.identity_user`/`server create` resource run).

All requests to Nova API, which need to talk to Neutron (I've been testing `server create`) will throw error:
<class 'nova.exception.NeutronAdminCredentialConfigurationInvalid'> (HTTP 500) (Request-ID: req-bb917e0c-0240-47a0-8485-20cedba059a8

Nova breaks:
* is_metal 25.2.0
* is_metal 25.2.0 with Nova from Zed (9bca7f34)
* is_metal without using uwsgi for nova-api-os-compute
* lxc aio 25.2.0 with
---
nova_wsgi_threads: 1
nova_wsgi_processes_max: 16
nova_wsgi_processes: 16

Nova works:
* lxc aio stable/zed
* is_metal 25.2.0 with Nova from Zed (9bca7f34) and uwsgi `processes = 1`
* is_metal 25.2.0 with Nova from Zed (9bca7f34) and uwsgi `processes = 1` and `threads > 1`
* lxc aio stable/yoga

It seems that recent fixes for OSSA-2023-003 (service user token stuff) almost fixed this problem, as with them it takes very accurate timing to reproduce `nova.exception.NeutronAdminCredentialConfigurationInvalid` (creating server during password update) and Nova itself works fine on subsequent requests.

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-plugins (master)
Changed in openstack-ansible:
status: New → In Progress
Changed in openstack-ansible:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-plugins (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888153
Committed: https://opendev.org/openstack/openstack-ansible-plugins/commit/088b6a1e523bb6cc477b1a3271d3b08bbff03b91
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 088b6a1e523bb6cc477b1a3271d3b08bbff03b91
Author: Dmitriy Rabotyagov <email address hidden>
Date: Tue Jun 20 13:44:18 2023 +0200

    Skip updating service password by default

    At the moment we always do attempt to reset passwords for the
    keystone services, which in some cases leads to race conditions in
    services. Thus, running a role is not idempotent which we fix by
    introducing a `service_update_password` variable. So whenever password
    needs to be reseted/updated, the variable should be supplied for that.

    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/888437
    Change-Id: I11b1046ea91cef7de0b2f6433baabbb144e07700
    Closes-Bug: #2023370
    (cherry picked from commit f35126af68e17d76be00f1cb70cd42fab15f2f4e)

tags: added: in-stable-zed
Changed in openstack-ansible:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.