Unit tests intermittently fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" during service destroy

Bug #1851871 reported by Matt Riedemann on 2019-11-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned

Bug Description

Seen here:

https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e8/691900/3/check/openstack-tox-py27/6e8b215/testr_results.html.gz

ft1.34: nova.tests.unit.conductor.test_conductor.ConductorTaskAPITestCase.test_unshelve_instance_schedule_and_rebuild_novalid_hosttesttools.testresult.real._StringException: pythonlogging:'': {{{
2019-11-08 16:07:22,725 INFO [nova.virt.driver] Loading compute driver 'fake.SmallFakeDriver'
2019-11-08 16:07:22,735 INFO [nova.compute.manager] Deleting orphan compute node 2 hypervisor host is fake_phyp1, nodes are set(['fakenode1', 'fakenode2'])
2019-11-08 16:07:22,739 ERROR [nova.compute.manager] Failed to delete compute node resource provider for compute node 906973a2-d73c-4254-8f1b-dfce1ae55359: An auth plugin is required to determine endpoint URL
2019-11-08 16:07:22,753 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode1
2019-11-08 16:07:22,760 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode1 with uuid: f4a97ca1-8388-4005-a3e0-d714beebcd69
2019-11-08 16:07:22,803 WARNING [nova.compute.resource_tracker] No compute node record for fake-mini:fakenode2
2019-11-08 16:07:22,807 INFO [nova.compute.resource_tracker] Compute node record created for fake-mini:fakenode2 with uuid: bbd1fd88-6b11-4de7-9d0e-a909d51df43d
2019-11-08 16:07:22,851 INFO [nova.service] Starting conductor node (version 20.1.0)
2019-11-08 16:07:22,970 WARNING [nova.conductor.manager] No valid host found for unshelve instance
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/fixture.py", line 125, in cleanUp
    return self._cleanups(raise_errors=raise_first)
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 89, in __call__
    reraise(error[0], error[1], error[2])
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/fixtures/callmany.py", line 83, in __call__
    cleanup(*args, **kwargs)
  File "nova/service.py", line 271, in kill
    self.service_ref.destroy()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
    return fn(self, *args, **kwargs)
  File "nova/objects/service.py", line 446, in destroy
    db.service_destroy(self._context, self.id)
  File "nova/db/api.py", line 99, in service_destroy
    return IMPL.service_destroy(context, service_id)
  File "nova/db/sqlalchemy/api.py", line 222, in wrapped
    with ctxt_mgr.writer.using(context):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
    context=context) as resource:
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 710, in _produce_block
    self._writer()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 725, in _writer
    "Can't upgrade a READER transaction "
TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction

But there are a few hits for this in other tests/jobs for unrelated changes:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22TypeError%3A%20Can't%20upgrade%20a%20READER%20transaction%20to%20a%20WRITER%20mid-transaction%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d

12 fails in 7 days.

Matt Riedemann (mriedem) on 2019-11-08
summary: - Functional tests intermittently fail with "TypeError: Can't upgrade a
- READER transaction to a WRITER mid-transaction" during service destroy
+ Unit tests intermittently fail with "TypeError: Can't upgrade a READER
+ transaction to a WRITER mid-transaction" during service destroy
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers