Unhandled error: RuntimeError: dictionary changed size during iteration

[Michal Nasiadka]

Observed in CentOS8 nova-scheduler logs:

2020-06-23 13:37:46.779 23 ERROR nova Traceback (most recent call last):
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module>
2020-06-23 13:37:46.779 23 ERROR nova sys.exit(main())
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main
2020-06-23 13:37:46.779 23 ERROR nova service.serve(server, workers=workers)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 489, in serve
2020-06-23 13:37:46.779 23 ERROR nova restart_method='mutate')
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch
2020-06-23 13:37:46.779 23 ERROR nova launcher.launch_service(service, workers=workers)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service
2020-06-23 13:37:46.779 23 ERROR nova self._start_child(wrap)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child
2020-06-23 13:37:46.779 23 ERROR nova self.launcher.restart()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart
2020-06-23 13:37:46.779 23 ERROR nova self.conf.mutate_config_files()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files
2020-06-23 13:37:46.779 23 ERROR nova self._warn_immutability()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability
2020-06-23 13:37:46.779 23 ERROR nova for info, group in self._all_opt_infos():
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos
2020-06-23 13:37:46.779 23 ERROR nova for info in self._opts.values():
2020-06-23 13:37:46.779 23 ERROR nova RuntimeError: dictionary changed size during iteration
2020-06-23 13:37:46.779 23 ERROR nova
2020-06-23 13:37:46.780 20 CRITICAL nova [req-739357c3-918b-4778-ab8b-282fc7fad943 - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration

OpenStack installed using RDO packages
Package versions:
(nova-scheduler)[nova@spsrc-controller-2 /]$ rpm -vqa | grep nova
python3-nova-20.2.0-1.el8.noarch
python3-novaclient-15.1.0-1.el8.noarch
openstack-nova-common-20.2.0-1.el8.noarch
openstack-nova-scheduler-20.2.0-1.el8.noarch
(nova-scheduler)[nova@spsrc-controller-2 /]$ rpm -vqa | grep oslo
python-oslo-utils-lang-3.41.5-1.el8.noarch
python-oslo-i18n-lang-3.24.0-2.el8.noarch
python-oslo-cache-lang-1.37.0-2.el8.noarch
python3-oslo-concurrency-3.30.0-2.el8.noarch
python3-oslo-messaging-10.2.0-2.el8.noarch
python-oslo-versionedobjects-lang-1.36.1-1.el8.noarch
python-oslo-policy-lang-2.3.3-1.el8.noarch
python-oslo-log-lang-3.44.2-1.el8.noarch
python3-oslo-i18n-3.24.0-2.el8.noarch
python-oslo-concurrency-lang-3.30.0-2.el8.noarch
python3-oslo-serialization-2.29.2-2.el8.noarch
python3-oslo-config-6.11.2-1.el8.noarch
python3-oslo-log-3.44.2-1.el8.noarch
python3-oslo-service-1.40.2-2.el8.noarch
python3-oslo-middleware-3.38.1-2.el8.noarch
python-oslo-vmware-lang-2.34.1-1.el8.noarch
python3-oslo-privsep-1.33.3-1.el8.noarch
python3-oslo-vmware-2.34.1-1.el8.noarch
python3-oslo-db-5.0.2-2.el8.noarch
python3-oslo-policy-2.3.3-1.el8.noarch
python3-oslo-reports-1.30.0-1.el8.noarch
python3-oslo-rootwrap-5.16.1-1.el8.noarch
python-oslo-privsep-lang-1.33.3-1.el8.noarch
python-oslo-middleware-lang-3.38.1-2.el8.noarch
python-oslo-db-lang-5.0.2-2.el8.noarch
python3-oslo-utils-3.41.5-1.el8.noarch
python3-oslo-context-2.23.0-2.el8.noarch
python3-oslo-cache-1.37.0-2.el8.noarch
python3-oslo-versionedobjects-1.36.1-1.el8.noarch
python3-oslo-upgradecheck-0.3.2-1.el8.noarch

[Michal Nasiadka (mnasiadka)]

Observation - this happens during stopping nova-scheduler.

OpenStack release: Train

[Balazs Gibizer (balazs-gibizer)]

With a stable/train devstack I cannot reproduce the above stack trace by stopping or restarting the nova-scheduler.

The relevant oslo libs are at the same version in my env as in your and I don't see any relevant changes between nova 20.2.0 and stable/train HEAD. I hope oslo folks will have some idea as without reproduction I cannot really debug this further. Marking this as incomplete from Nova perspective until more information is provided. Feel free to set it back to New later.

[Takashi Kajinami (kajinamit)]

I'm closing this because of inactivity. I've never seen this problem, either.