nova-manage db archive_deleted_rows crash on tags table

Bug #1782328 reported by Krzysztof Pawlowski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Undecided
Zhenyu Zheng

Bug Description

Description
===========
nova-manage db archive_deleted_rows crash during processing of nova.tags table because table do not have `id` column.

Steps to reproduce
==================
Execute: nova-manage db archive_deleted_rows

Expected result
===============
Successful execution with summary:

+-----------------+-------------------------+
| Table | Number of Rows Archived |
+-----------------+-------------------------+
| agent_builds | 12 |
| floating_ips | 12 |
| instance_groups | 7 |
| instance_types | 6 |
| key_pairs | 44242 |
+-----------------+-------------------------+

Actual result
=============
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1644, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 532, in archive_deleted_rows
    run, deleted_instance_uuids = db.archive_deleted_rows(max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 2018, in archive_deleted_rows
    return IMPL.archive_deleted_rows(max_rows=max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 6233, in archive_deleted_rows
    tablename, max_rows=max_rows - total_rows_archived)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 6116, in _archive_deleted_rows_for_table
    column = table.c.id
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
    raise AttributeError(key)
AttributeError: id

Environment
===========
Openstack Queens release:
openstack-nova-conductor-17.0.5-1.el7.noarch
openstack-nova-scheduler-17.0.5-1.el7.noarch
openstack-nova-api-17.0.5-1.el7.noarch
openstack-nova-placement-api-17.0.5-1.el7.noarch
openstack-nova-novncproxy-17.0.5-1.el7.noarch
openstack-nova-console-17.0.5-1.el7.noarch
openstack-nova-common-17.0.5-1.el7.noarch

Tags: db nova-manage
description: updated
Matt Riedemann (mriedem)
tags: added: nova-manage
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

I cannot reproduce this as when instance got destoried the related tags are also deleted:
https://github.com/openstack/nova/blob/683fbe3c84ee70e817ffb8396fefb9ff6cd8830d/nova/db/sqlalchemy/api.py#L1824
and there are no deleted field in the tags table, so their should be nothing to archieve in the tags table when running this command.

Revision history for this message
Krzysztof Pawlowski (krzysztof.pawlowski) wrote :

I my environment 'tags' table is empty and during nova-manage db archive_deleted_rows I've got above exception.

In my opinion it's because of:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6115
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6118

Table 'tags' do not have column 'id' and 'deleted' which are required by code above.

As I understand:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6219
all tables are processed except:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6224

As I have written above issue occurs on Queens release. Probably it's fixed on master branch.
It would be nice to have this also fixed on Queens.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Hmm, we run archive_deleted_rows in our CI so if this was a 100% failure I'm not sure why we wouldn't see it there.

http://logs.openstack.org/36/582536/1/check/nova-next/0852de7/logs/devstack-gate-post_test_hook.txt.gz#_2018-07-16_14_26_31_580

Do you have custom changes to nova?

Revision history for this message
Matt Riedemann (mriedem) wrote :

You should be hitting this short-circuit:

https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6106

Do you have a shadow_tags table?

Changed in nova:
status: New → Incomplete
tags: added: db
Revision history for this message
Krzysztof Pawlowski (krzysztof.pawlowski) wrote :

shadow_tags table exists:

       Table: shadow_tags
Create Table: CREATE TABLE `shadow_tags` (
  `resource_id` varchar(36) NOT NULL,
  `tag` varchar(80) NOT NULL,
  PRIMARY KEY (`resource_id`,`tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

According to our changes history this table appeared after upgrade to Ocata release.

Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

Hi, I did not saw the above mentioned code in Ocata, Pike and Queens code, where actually is that code?

Revision history for this message
Krzysztof Pawlowski (krzysztof.pawlowski) wrote :

I'm not able trace piece of code which was responsible for shadow_tags table creation but somehow it happened during upgrade. I will try to reproduce it during:
- upgrade from newton to ocata
- fresh ocata installation
- fresh pike installation
- fresh queens installation

Revision history for this message
Krzysztof Pawlowski (krzysztof.pawlowski) wrote :

I've performed fresh installation using packstack of:
- newton
- ocata
- pike
- queens
None of aboved have shadow_tags table in nova database.

I've also performed db upgrade from newton to ocata and shadow_tags table was not introduced.

I don't know how to reproduce it although shadow_tags table appeared in our two openstack clouds after upgrade from newton to ocata.

I solved issue by deleting shadow_tags table.

Revision history for this message
Daniel Speichert (dasp) wrote :

I ran into the same issue:

# nova-manage db archive_deleted_rows --verbose --until-complete
 Use option "connection_recycle_time" from group "api_database".
Archiving..An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 2300, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 542, in archive_deleted_rows
    run, deleted_instance_uuids = db.archive_deleted_rows(max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1802, in archive_deleted_rows
    return IMPL.archive_deleted_rows(max_rows=max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5581, in archive_deleted_rows
    tablename, max_rows=max_rows - total_rows_archived)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5467, in _archive_deleted_rows_for_table
    column = table.c.id
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
    raise AttributeError(key)
AttributeError: id

Dropping shadow_tags solved the problem. The database was upgraded in-place at least since Juno/Kilo.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.