Comment 4 for bug 1746509

Hello Saulo,

I run in the very same problem doing the very same update as you (Ocata --> Pike).

I'm not sure the reason is the same, but I can show you the way i fixed (workarounded?) it, maybe it could help you too.

The reason the error:"Can't upgrade a READER transaction to a WRITE.." comes up can be found in the following lines of the error log you provided:

... ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object
... ERROR nova.api.openstack.extensions service.save()

The _from_db_object functions (module service.py) iterates over the service object and verifies if it contains the uuid, in case the uuid is missing a new one get generated and saved.

246 if 'uuid' not in service:
247 service.uuid = uuidutils.generate_uuid()
248 LOG.debug('Generated UUID %(uuid)s for service %(id)i',
249 dict(uuid=service.uuid, id=service.id))
250 service.save()
251
252 return service

But doing the save (a WRITE) on a context open for READ generates the above error (Can't upgrade a READER transaction to a WRITE..).

To workaround this I patched this module this way:

29 from nova.context import RequestContext
...
239 # service._context = context
240 service._context = RequestContext.from_dict(context.to_dict())
241 service.obj_reset_changes()
242
243 # TODO(dpeschman): Drop this once all services have uuids in database
244 if 'uuid' not in service:
245 service.uuid = uuidutils.generate_uuid()
246 LOG.debug('Generated UUID %(uuid)s for service %(id)i',
247 dict(uuid=service.uuid, id=service.id))
248 service.save()

This way you should not actually solve the problem but It should help you expose the real one.

In may case it showed up the following:

2018-02-08 15:43:13.088 9878 ERROR nova.api.openstack.extensions ServiceNotFound: Service 7 could not be found.

Looking in the service table in the nova DB i found this:

MariaDB [nova]> select * from services;
+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
| 2017-03-21 17:02:32 | 2018-02-08 15:07:47 | NULL | 1 | controller | nova-consoleauth | consoleauth | 2646887 | 0 | 0 | NULL | 2018-02-08 15:07:47 | 0 | 22 | 1adc00a7-177c-49b3-bd92-9c9f51953d07 |
| 2017-03-21 17:02:32 | 2018-02-08 15:07:47 | NULL | 2 | controller | nova-scheduler | scheduler | 2646942 | 0 | 0 | NULL | 2018-02-08 15:07:47 | 0 | 22 | 2d087a72-44b6-4250-8a9c-3d0576da399d |
| 2017-03-21 17:02:32 | 2018-02-08 15:07:48 | NULL | 3 | controller | nova-conductor | conductor | 2647664 | 0 | 0 | NULL | 2018-02-08 15:07:48 | 0 | 22 | fb8ce489-ece5-4b58-81b0-b89c73bb458a |
| 2017-03-21 17:02:33 | 2018-02-08 14:40:39 | NULL | 4 | 0.0.0.0 | nova-osapi_compute | NULL | 0 | 0 | 0 | NULL | NULL | 0 | 22 | 334c6402-f74e-4046-8a23-20511dcaecfc |
| 2017-03-21 17:02:35 | 2018-02-08 14:40:39 | NULL | 6 | 0.0.0.0 | nova-metadata | NULL | 0 | 0 | 0 | NULL | NULL | 0 | 22 | 5b4587b9-725a-42ed-8d86-4e5db129c6c7 |
| 2017-03-21 17:03:23 | 2017-09-07 15:03:01 | 2017-09-07 15:03:02 | 7 | vcentergw | nova-compute | compute | 1466363 | 0 | 7 | set by vmwareapi host_state | 2017-09-07 15:03:01 | 0 | 16 | NULL |
| 2017-03-21 17:05:16 | 2018-02-08 14:32:50 | NULL | 8 | compute1 | nova-compute | compute | 2544210 | 0 | 0 | NULL | 2018-02-06 14:08:33 | 0 | 16 | e0575f3c-9f0c-4ef2-94be-2b5ee8631d64 |
| 2017-03-21 17:06:39 | 2018-02-08 14:32:50 | NULL | 9 | compute2 | nova-compute | compute | 2646709 | 0 | 0 | NULL | 2018-02-06 14:08:34 | 0 | 16 | 9f0316da-5885-46ef-a07d-65dbe99f76dd |
| 2017-03-23 10:12:41 | 2018-02-08 14:32:50 | NULL | 10 | controller | nova-cert | cert | 2631642 | 0 | 0 | NULL | 2018-02-06 14:08:15 | 0 | 16 | db9ccde3-30c2-43e0-b3c7-1de34687c59e |
| 2017-09-07 15:03:59 | 2017-09-08 13:28:31 | 2017-09-08 13:28:37 | 11 | vcentergw | nova-compute | compute | 8047 | 0 | 11 | NULL | 2017-09-08 13:28:31 | 0 | 16 | NULL |
| 2017-09-08 13:32:38 | 2018-02-08 15:07:46 | NULL | 12 | vcentergw | nova-compute | compute | 1171103 | 0 | 0 | set by vmwareapi host_state | 2018-02-08 15:07:46 | 0 | 22 | 57144637-d6e3-4492-b6a4-2dff3485b61d |
| 2017-12-14 15:23:38 | 2018-02-08 14:32:50 | NULL | 13 | compute5 | nova-compute | compute | 231136 | 0 | 0 | NULL | 2018-01-10 09:45:35 | 0 | 16 | a5906a88-25a1-4d69-8b10-51a28a00faf8 |
| 2017-12-14 15:23:38 | 2018-02-08 14:32:50 | NULL | 14 | compute3 | nova-compute | compute | 305003 | 0 | 0 | NULL | 2018-02-04 14:02:24 | 0 | 16 | ccc58892-8bf1-4738-b64f-5f58268f73a9 |
| 2018-01-16 15:37:04 | 2018-02-08 15:07:52 | NULL | 15 | vcentergw2 | nova-compute | compute | 180952 | 0 | 0 | NULL | 2018-02-08 15:07:52 | 0 | 22 | 62822ffd-a8c4-4ac5-9610-e8612d8aec3b |
+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
14 rows in set (0.00 sec)

There was the service compute-agent from the previous version still registered without UUID and with the field version to 16.

+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+

| 2017-03-21 17:03:23 | 2017-09-07 15:03:01 | 2017-09-07 15:03:02 | 7 | vcentergw | nova-compute | compute | 1466363 | 0 | 7 | set by vmwareapi host_state | 2017-09-07 15:03:01 | 0 | 16 | NULL

So I simply removed from the DB the service from the old version:

MariaDB [nova]> delete from services where version=16;
Query OK, 7 rows affected (0.02 sec)

And restarted nova-api.

At this point the instances list (even the old ones) show up.

Dunno if exists a simplier way to clean up the DB after an upgrade.

Hope it helps.

Bye.