After Newton->Ocata undercloud upgrade the overcloud nova nodes end up in ERROR state

[Marius Cornea]

After Newton->Ocata undercloud upgrade the overcloud nova nodes end up in ERROR state:

[stack@undercloud-0 ~]$ nova list
/usr/lib/python2.7/site-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+--------------------------------------+-------------------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+-----------------------+
| f76c4722-c649-4b80-b629-4e6afb8e73d2 | overcloud-cephstorage-0 | ERROR | - | Running | ctlplane=192.168.0.17 |
| 4c6d8498-0acd-4975-9c09-aa15d5cca35d | overcloud-cephstorage-1 | ERROR | - | Running | ctlplane=192.168.0.20 |
| 18c33a92-cace-444c-86f0-ad666fadc69e | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.0.11 |
| d7911872-d5f2-46a4-a965-c600b4228b2e | overcloud-controller-1 | ERROR | - | Running | ctlplane=192.168.0.21 |
| 3f1dbe42-0d4e-403a-a979-68d4c730b7d7 | overcloud-controller-2 | ERROR | - | Running | ctlplane=192.168.0.26 |
| 8d8e18df-3c77-4907-a7b2-fd0bc0051a88 | overcloud-novacompute-0 | ERROR | - | Running | ctlplane=192.168.0.23 |
| 114ecb95-873d-4aa1-9d5a-71506bce6524 | overcloud-novacompute-1 | ERROR | - | Running | ctlplane=192.168.0.19 |
+--------------------------------------+-------------------------+--------+------------+-------------+-----------------------+

[Marius Cornea (mcornea)]

Attaching the nova logs.

[Brent Eagles (beagles)]

At least some of the supplied logs indicate some incompatibilities between code and data. e.g. InstanceList 2.2 not supported. I wonder if this could be caused by upgrading the undercloud without stopping the services first?

[Marius Cornea (mcornea)]

Yes, it looks so. I wasn't able to reproduce the issue when stopping the services before running openstack undercloud upgrade.

[Emilien Macchi (emilienm)]

It is documentere here:
http://tripleo.org/installation/installation.html#updating-undercloud-components

that services need to be stopped before running the upgrade. Closing the bug.