multinode setup, devstack scheduler fails to start after controller restart

[Amit Uniyal]

In multinode devstack setup nova scheduler fails to start after reboot

Steps to reproduce
==================

1 - deploy multinode devstack
    https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

2 - Verify all compute nodes are listed and setup is working as expected
    $ openstack compute service list

    create vm, assign floating IP and access VM

3 - Restart compute nodes, and controller node
    $ sudo init 6

4 - Once controller and all other nodes are rebooted, check whether all nova services are running
    $ openstack compute service list

    $ sudo systemctl status devstack@n-*

Expected result
===============
$ sudo systemctl status devstack@n-*

All services should be running

$ openstack compute service list

openstack cmds should run without a issue,

Actual result
=============
nova-schduler fails to start with error:
============================================
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova self._init_plugins(extensions)
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova raise NoMatches('No %r driver found, looking for %r' %
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING stevedore.named [-] Could not load filter_scheduler
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova [-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova Traceback (most recent call last):
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/bin/nova-scheduler", line 10, in <module>
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova sys.exit(main())
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova server = service.Service.create(binary='nova-scheduler',
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/opt/stack/nova/nova/service.py", line 252, in create
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova service_obj = cls(host, binary, topic, manager,
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/opt/stack/nova/nova/service.py", line 116, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova self.driver = driver.DriverManager(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova

============================================

$ openstack compute service list
HttpException: 500: Server Error for url: http://22.0.2.5/compute/v2.1/os-services, Internal Server Error

$ sudo systemctl status devstack@n-sch
● <email address hidden> - Devstack <email address hidden>
     Loaded: loaded (/<email address hidden>; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC; 3min 34s ago
    Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file /etc/nova/nova.conf (code=exited, status=1/FAILURE)
   Main PID: 11226 (code=exited, status=1/FAILURE)

Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in _init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: <email address hidden>: Failed with result 'exit-code'.

Environment
===========
Devstack multinode setup

1 controller
2 computes

[Amit Uniyal (auniyal)]

redeployment fixes the issue,
./unstack.sh
./clean.sh
./stack.sh

But its time consuming,

[Sylvain Bauza (sylvain-bauza)]

Looks to me not a Nova issue, maybe just a devstack issue or a configuration problem. Moving it then to devstack.

[Dr. Jens Harbott (j-harbott)]

Devstack isn't meant to be rebooted. If you can come up with a patch to improve this issue, we will review it, but otherwise redeploying after a reboot is the expected solution.