Unhandled error: RuntimeError: dictionary changed size during iteration

Bug #1884762 reported by Michal Nasiadka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
oslo.config
Invalid
Undecided
Unassigned

Bug Description

Observed in CentOS8 nova-scheduler logs:

2020-06-23 13:37:46.779 23 ERROR nova Traceback (most recent call last):
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module>
2020-06-23 13:37:46.779 23 ERROR nova sys.exit(main())
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main
2020-06-23 13:37:46.779 23 ERROR nova service.serve(server, workers=workers)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 489, in serve
2020-06-23 13:37:46.779 23 ERROR nova restart_method='mutate')
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch
2020-06-23 13:37:46.779 23 ERROR nova launcher.launch_service(service, workers=workers)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service
2020-06-23 13:37:46.779 23 ERROR nova self._start_child(wrap)
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child
2020-06-23 13:37:46.779 23 ERROR nova self.launcher.restart()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart
2020-06-23 13:37:46.779 23 ERROR nova self.conf.mutate_config_files()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files
2020-06-23 13:37:46.779 23 ERROR nova self._warn_immutability()
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability
2020-06-23 13:37:46.779 23 ERROR nova for info, group in self._all_opt_infos():
2020-06-23 13:37:46.779 23 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos
2020-06-23 13:37:46.779 23 ERROR nova for info in self._opts.values():
2020-06-23 13:37:46.779 23 ERROR nova RuntimeError: dictionary changed size during iteration
2020-06-23 13:37:46.779 23 ERROR nova
2020-06-23 13:37:46.780 20 CRITICAL nova [req-739357c3-918b-4778-ab8b-282fc7fad943 - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration

OpenStack installed using RDO packages
Package versions:
(nova-scheduler)[nova@spsrc-controller-2 /]$ rpm -vqa | grep nova
python3-nova-20.2.0-1.el8.noarch
python3-novaclient-15.1.0-1.el8.noarch
openstack-nova-common-20.2.0-1.el8.noarch
openstack-nova-scheduler-20.2.0-1.el8.noarch
(nova-scheduler)[nova@spsrc-controller-2 /]$ rpm -vqa | grep oslo
python-oslo-utils-lang-3.41.5-1.el8.noarch
python-oslo-i18n-lang-3.24.0-2.el8.noarch
python-oslo-cache-lang-1.37.0-2.el8.noarch
python3-oslo-concurrency-3.30.0-2.el8.noarch
python3-oslo-messaging-10.2.0-2.el8.noarch
python-oslo-versionedobjects-lang-1.36.1-1.el8.noarch
python-oslo-policy-lang-2.3.3-1.el8.noarch
python-oslo-log-lang-3.44.2-1.el8.noarch
python3-oslo-i18n-3.24.0-2.el8.noarch
python-oslo-concurrency-lang-3.30.0-2.el8.noarch
python3-oslo-serialization-2.29.2-2.el8.noarch
python3-oslo-config-6.11.2-1.el8.noarch
python3-oslo-log-3.44.2-1.el8.noarch
python3-oslo-service-1.40.2-2.el8.noarch
python3-oslo-middleware-3.38.1-2.el8.noarch
python-oslo-vmware-lang-2.34.1-1.el8.noarch
python3-oslo-privsep-1.33.3-1.el8.noarch
python3-oslo-vmware-2.34.1-1.el8.noarch
python3-oslo-db-5.0.2-2.el8.noarch
python3-oslo-policy-2.3.3-1.el8.noarch
python3-oslo-reports-1.30.0-1.el8.noarch
python3-oslo-rootwrap-5.16.1-1.el8.noarch
python-oslo-privsep-lang-1.33.3-1.el8.noarch
python-oslo-middleware-lang-3.38.1-2.el8.noarch
python-oslo-db-lang-5.0.2-2.el8.noarch
python3-oslo-utils-3.41.5-1.el8.noarch
python3-oslo-context-2.23.0-2.el8.noarch
python3-oslo-cache-1.37.0-2.el8.noarch
python3-oslo-versionedobjects-1.36.1-1.el8.noarch
python3-oslo-upgradecheck-0.3.2-1.el8.noarch

Tags: oslo scheduler
Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

Observation - this happens during stopping nova-scheduler.

OpenStack release: Train

tags: added: oslo scheduler
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

With a stable/train devstack I cannot reproduce the above stack trace by stopping or restarting the nova-scheduler.

The relevant oslo libs are at the same version in my env as in your and I don't see any relevant changes between nova 20.2.0 and stable/train HEAD. I hope oslo folks will have some idea as without reproduction I cannot really debug this further. Marking this as incomplete from Nova perspective until more information is provided. Feel free to set it back to New later.

Changed in nova:
status: New → Incomplete
Revision history for this message
Takashi Kajinami (kajinamit) wrote :

I'm closing this because of inactivity. I've never seen this problem, either.

Changed in oslo.config:
status: New → Invalid
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.