Variety of Heat-related Tempest tests fails with OperationalError

Bug #1620638 reported by Anna Babich
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Fix Committed
High
Maksim Malchuk

Bug Description

Environment where it has been reproduced:
ababich@srv138-bud:~$ grep SNAPSHOT /home/jenkins/env_inject.properties
SNAPSHOT_ID=#225
ababich@srv138-bud:~$

[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+--------+---------------------------+---------+------------+-------------------+-------------------+---------------+--------+---------
 3 | ready | slave-01_controller_mongo | 1 | 10.109.2.3 | 64:90:06:c0:b7:2f | controller, mongo | | 1 | 1
 6 | ready | slave-06_ironic | 1 | 10.109.2.8 | 64:7f:6a:12:52:53 | ironic | | 1 | 1
 1 | ready | slave-03_controller_mongo | 1 | 10.109.2.5 | 64:88:73:ce:0c:b9 | controller, mongo | | 1 | 1
 5 | ready | slave-02_controller_mongo | 1 | 10.109.2.4 | 64:33:cf:78:73:4a | controller, mongo | | 1 | 1
 2 | ready | slave-05_compute_cinder | 1 | 10.109.2.7 | 64:99:c9:0a:2f:09 | cinder, compute | | 1 | 1
 4 | ready | slave-04_compute_cinder | 1 | 10.109.2.6 | 64:80:b7:c4:e9:43 | cinder, compute | | 1 | 1

About 20 Heat-related Tempest tests (http://paste.openstack.org/show/567167/) failed with an error which looks like:
tempest.lib.exceptions.ServerFault: Got server fault

Details: Remote error: OperationalError (_mysql_exceptions.OperationalError) (1054, "Unknown column 'stack.parent_resource_name' in 'field list'") [SQL: u'SELECT stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.status_reason AS stack_status_reason, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.prev_raw_template_id AS stack_prev_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.parent_resource_name AS stack_parent_resource_name, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.nested_depth AS stack_nested_depth, stack.convergence AS stack_convergence, stack.current_traversal AS stack_current_traversal, stack.current_deps AS stack_current_deps, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND (stack.tenant = %s OR stack.stack_user_project_id = %s) AND stack.name = %s \n LIMIT %s'] [parameters: ('6f5e72831a9e4454836f1fb3df0a1cde', '6f5e72831a9e4454836f1fb3df0a1cde', 'tempest-heat-374357048', 1)]

[u'

Also, when trying to run 'heat stack-list' against controller belonging to indicated env, the result is:
root@node-1:~# heat stack-list
ERROR: Remote error: OperationalError (_mysql_exceptions.OperationalError) (1054, "Unknown column 'stack.parent_resource_name' in 'field list'") [SQL: u'SELECT stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.status_reason AS stack_status_reason, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.prev_raw_template_id AS stack_prev_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.parent_resource_name AS stack_parent_resource_name, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.nested_depth AS stack_nested_depth, stack.convergence AS stack_convergence, stack.current_traversal AS stack_current_traversal, stack.current_deps AS stack_current_deps, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = %s AND NOT (EXISTS (SELECT 1 \nFROM stack_tag \nWHERE stack.id = stack_tag.stack_id AND stack_tag.tag IN (%s))) ORDER BY stack.created_at DESC, stack.id DESC'] [parameters: ('fb384b84ca984f52849359166573bd1b', 'data-processing-cluster')]
[u'

At the same time, everything is OK with listed-above tests when running them against another envs based on the same snapshot. Looks like specific env's failure.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

It looks like we have several really similar bugs (not duplicates!) - because yesterday I saw the same behavior for Nova service:

https://bugs.launchpad.net/fuel/+bug/1619740

Changed in mos:
assignee: MOS QA Team (mos-qa) → MOS Heat (mos-heat)
status: New → Confirmed
tags: added: area-heat
Changed in mos:
assignee: MOS Heat (mos-heat) → Peter Razumovsky (prazumovsky)
Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

First, could you share syslog lines which contain "mysql_status()" pattern?

Second, seems like it's not (only) heat problem: similar behaviour observed in Murano (https://bugs.launchpad.net/fuel/+bug/1620268), cinder and neutron (https://bugs.launchpad.net/fuel/+bug/1620268/comments/6).

Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

And third, does this issue reproduced repeatedly or failed only once?

Revision history for this message
Anna Babich (ababich) wrote :

The env where the issue happened was deployed by CI daily job and has been already destroyed, so we don't have extra logs. As you can see from bug's description, "At the same time, everything is OK with listed-above tests when running them against another envs based on the same snapshot. Looks like specific env's failure."
And these tests also passed against the latest snapshot for the same job where they failed before:
http://cz7776.bud.mirantis.net:8080/jenkins/job/9.x_Tempest_LVM_SSL/77/testReport/

Revision history for this message
Anna Babich (ababich) wrote :

Looks like it's expected to be fixed by patch proposed here: https://bugs.launchpad.net/fuel/+bug/1619740

Changed in mos:
assignee: Peter Razumovsky (prazumovsky) → Maksim Malchuk (mmalchuk)
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/#/c/367472/

Fix proposed to branch: master
Review: https://review.openstack.org/#/c/367474/

tags: added: area-library
tags: added: on-verification
no longer affects: mos/9.x
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.1 snapshot #285.

tags: removed: on-verification
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.