Storing events can fail (again) due to oversized properties

Bug #1671043 reported by Steven Hardy on 2017-03-08
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Invalid
High
Thomas Herve

Bug Description

We previously worked around problems where very large resource_properties can overflow the DB column causing errors like:

https://gist.github.com/jtaleric/6dec8d19a0eb591cd3e9dee1262b0b08

However this happened on a recent heat version (ocata) and I believe it's related to this change:

https://github.com/openstack/heat/commit/57c5aae88b908002d84a933738293c5917926174

Which AFAICS inadvertently removed the truncation logic we previously used?

We probably need to re-add equivalent logic, and reinstate a test which proves this can't happen again

Thomas Herve (therve) on 2017-03-08
Changed in heat:
importance: Undecided → High
assignee: nobody → Thomas Herve (therve)
milestone: none → pike-1
Steven Hardy (shardy) wrote :
Download full text (3.9 KiB)

One thing I did notice too is we've got a possible mismatch between longtext for resource.properties_data and blob (not longblob) for event.resource_properties:

MariaDB [heat]> describe resource_properties;
ERROR 1146 (42S02): Table 'heat.resource_properties' doesn't exist
MariaDB [heat]> describe resource;
+---------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uuid | varchar(36) | YES | UNI | NULL | |
| nova_instance | varchar(255) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| action | varchar(255) | YES | | NULL | |
| status | varchar(255) | YES | | NULL | |
| status_reason | longtext | YES | | NULL | |
| stack_id | varchar(36) | NO | MUL | NULL | |
| rsrc_metadata | longtext | YES | | NULL | |
| properties_data | longtext | YES | | NULL | |
| engine_id | varchar(36) | YES | | NULL | |
| atomic_key | int(11) | YES | | NULL | |
| needed_by | longtext | YES | | NULL | |
| requires | longtext | YES | | NULL | |
| replaces | int(11) | YES | | NULL | |
| replaced_by | int(11) | YES | | NULL | |
| current_template_id | int(11) | YES | MUL | NULL | |
| properties_data_encrypted | tinyint(1) | YES | | NULL | |
| root_stack_id | varchar(36) | YES | MUL | NULL | |
| rsrc_prop_data_id | int(11) | YES | MUL | NULL | |
+---------------------------+--------------+------+-----+---------+----------------+
22 rows in set (0.01 sec)

MariaDB [heat]> describe event;
+------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uuid | varchar(36) | YES | UNI | NULL | |
| stack_id | varchar(36) | NO | MUL | NULL | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES ...

Read more...

Changed in heat:
status: New → Triaged
importance: High → Critical
importance: Critical → High
Thomas Herve (therve) wrote :

OK, I looked back at the conversation, and this is newton AFAICT, so not a regression. Newton is missing https://review.openstack.org/#/c/381667/ which I fixed on Ocata, but didn't backport.

That said, this shouldn't be a fatal error anywhere. Except maybe for hooks, where it could trick the client.

Thomas Herve (therve) wrote :

That was not the issue.

Changed in heat:
status: Triaged → Invalid
Crag Wolfe (cwolfe) wrote :

As far as Ocata is conccerned, as long as it includes https://github.com/openstack/heat/commit/57c5aae88b908002d84a933738293c5917926174 (which it should), then the writing of new properties data for events or resources should always happen in the new resource_properties_data table which includes a longtext data column (and never in the legacy columns in resource or event). So, not sure how that stack trace could happen in Ocata.

I agree with therve that the change he linked to should fix the issue in Newton.

Joe Talerico (jtaleric) wrote :

This is Newton

Steven Hardy (shardy) wrote :

Ok thanks folks, apologies for the confusion re versions, it was indeed newton, so I cherry-picked the fix from therve ref https://review.openstack.org/#/c/443168/

Changed in heat:
milestone: pike-1 → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers