nova-manage db archive_deleted_rows crash on tags table

[Krzysztof Pawlowski]

Description
===========
nova-manage db archive_deleted_rows crash during processing of nova.tags table because table do not have `id` column.

Steps to reproduce
==================
Execute: nova-manage db archive_deleted_rows

Expected result
===============
Successful execution with summary:

+-----------------+-------------------------+
| Table | Number of Rows Archived |
+-----------------+-------------------------+
| agent_builds | 12 |
| floating_ips | 12 |
| instance_groups | 7 |
| instance_types | 6 |
| key_pairs | 44242 |
+-----------------+-------------------------+

Actual result
=============
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1644, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 532, in archive_deleted_rows
    run, deleted_instance_uuids = db.archive_deleted_rows(max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 2018, in archive_deleted_rows
    return IMPL.archive_deleted_rows(max_rows=max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 6233, in archive_deleted_rows
    tablename, max_rows=max_rows - total_rows_archived)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 6116, in _archive_deleted_rows_for_table
    column = table.c.id
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
    raise AttributeError(key)
AttributeError: id

Environment
===========
Openstack Queens release:
openstack-nova-conductor-17.0.5-1.el7.noarch
openstack-nova-scheduler-17.0.5-1.el7.noarch
openstack-nova-api-17.0.5-1.el7.noarch
openstack-nova-placement-api-17.0.5-1.el7.noarch
openstack-nova-novncproxy-17.0.5-1.el7.noarch
openstack-nova-console-17.0.5-1.el7.noarch
openstack-nova-common-17.0.5-1.el7.noarch

[Zhenyu Zheng (zhengzhenyu)]

I cannot reproduce this as when instance got destoried the related tags are also deleted:
https://github.com/openstack/nova/blob/683fbe3c84ee70e817ffb8396fefb9ff6cd8830d/nova/db/sqlalchemy/api.py#L1824
and there are no deleted field in the tags table, so their should be nothing to archieve in the tags table when running this command.

[Krzysztof Pawlowski (krzysztof.pawlowski)]

I my environment 'tags' table is empty and during nova-manage db archive_deleted_rows I've got above exception.

In my opinion it's because of:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6115
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6118

Table 'tags' do not have column 'id' and 'deleted' which are required by code above.

As I understand:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6219
all tables are processed except:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6224

As I have written above issue occurs on Queens release. Probably it's fixed on master branch.
It would be nice to have this also fixed on Queens.

[Matt Riedemann (mriedem)]

Hmm, we run archive_deleted_rows in our CI so if this was a 100% failure I'm not sure why we wouldn't see it there.

http://logs.openstack.org/36/582536/1/check/nova-next/0852de7/logs/devstack-gate-post_test_hook.txt.gz#_2018-07-16_14_26_31_580

Do you have custom changes to nova?

[Matt Riedemann (mriedem)]

You should be hitting this short-circuit:

https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6106

Do you have a shadow_tags table?

[Krzysztof Pawlowski (krzysztof.pawlowski)]

shadow_tags table exists:

       Table: shadow_tags
Create Table: CREATE TABLE `shadow_tags` (
  `resource_id` varchar(36) NOT NULL,
  `tag` varchar(80) NOT NULL,
  PRIMARY KEY (`resource_id`,`tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

According to our changes history this table appeared after upgrade to Ocata release.

[Zhenyu Zheng (zhengzhenyu)]

Hi, I did not saw the above mentioned code in Ocata, Pike and Queens code, where actually is that code?

[Krzysztof Pawlowski (krzysztof.pawlowski)]

I'm not able trace piece of code which was responsible for shadow_tags table creation but somehow it happened during upgrade. I will try to reproduce it during:
- upgrade from newton to ocata
- fresh ocata installation
- fresh pike installation
- fresh queens installation

[Krzysztof Pawlowski (krzysztof.pawlowski)]

I've performed fresh installation using packstack of:
- newton
- ocata
- pike
- queens
None of aboved have shadow_tags table in nova database.

I've also performed db upgrade from newton to ocata and shadow_tags table was not introduced.

I don't know how to reproduce it although shadow_tags table appeared in our two openstack clouds after upgrade from newton to ocata.

I solved issue by deleting shadow_tags table.

[Daniel Speichert (dasp)]

I ran into the same issue:

# nova-manage db archive_deleted_rows --verbose --until-complete
 Use option "connection_recycle_time" from group "api_database".
Archiving..An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 2300, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 542, in archive_deleted_rows
    run, deleted_instance_uuids = db.archive_deleted_rows(max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1802, in archive_deleted_rows
    return IMPL.archive_deleted_rows(max_rows=max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5581, in archive_deleted_rows
    tablename, max_rows=max_rows - total_rows_archived)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5467, in _archive_deleted_rows_for_table
    column = table.c.id
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
    raise AttributeError(key)
AttributeError: id

Dropping shadow_tags solved the problem. The database was upgraded in-place at least since Juno/Kilo.