Storing events can fail (again) due to oversized properties

[Steven Hardy]

We previously worked around problems where very large resource_properties can overflow the DB column causing errors like:

https://gist.github.com/jtaleric/6dec8d19a0eb591cd3e9dee1262b0b08

However this happened on a recent heat version (ocata) and I believe it's related to this change:

https://github.com/openstack/heat/commit/57c5aae88b908002d84a933738293c5917926174

Which AFAICS inadvertently removed the truncation logic we previously used?

We probably need to re-add equivalent logic, and reinstate a test which proves this can't happen again

[Steven Hardy (shardy)]

One thing I did notice too is we've got a possible mismatch between longtext for resource.properties_data and blob (not longblob) for event.resource_properties:

MariaDB [heat]> describe resource_properties;
ERROR 1146 (42S02): Table 'heat.resource_properties' doesn't exist
MariaDB [heat]> describe resource;
+---------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uuid | varchar(36) | YES | UNI | NULL | |
| nova_instance | varchar(255) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| action | varchar(255) | YES | | NULL | |
| status | varchar(255) | YES | | NULL | |
| status_reason | longtext | YES | | NULL | |
| stack_id | varchar(36) | NO | MUL | NULL | |
| rsrc_metadata | longtext | YES | | NULL | |
| properties_data | longtext | YES | | NULL | |
| engine_id | varchar(36) | YES | | NULL | |
| atomic_key | int(11) | YES | | NULL | |
| needed_by | longtext | YES | | NULL | |
| requires | longtext | YES | | NULL | |
| replaces | int(11) | YES | | NULL | |
| replaced_by | int(11) | YES | | NULL | |
| current_template_id | int(11) | YES | MUL | NULL | |
| properties_data_encrypted | tinyint(1) | YES | | NULL | |
| root_stack_id | varchar(36) | YES | MUL | NULL | |
| rsrc_prop_data_id | int(11) | YES | MUL | NULL | |
+---------------------------+--------------+------+-----+---------+----------------+
22 rows in set (0.01 sec)

MariaDB [heat]> describe event;
+------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| uuid | varchar(36) | YES | UNI | NULL | |
| stack_id | varchar(36) | NO | MUL | NULL | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES ...

[Thomas Herve (therve)]

OK, I looked back at the conversation, and this is newton AFAICT, so not a regression. Newton is missing https://review.openstack.org/#/c/381667/ which I fixed on Ocata, but didn't backport.

That said, this shouldn't be a fatal error anywhere. Except maybe for hooks, where it could trick the client.

[Thomas Herve (therve)]

That was not the issue.

[Crag Wolfe (cwolfe)]

As far as Ocata is conccerned, as long as it includes https://github.com/openstack/heat/commit/57c5aae88b908002d84a933738293c5917926174 (which it should), then the writing of new properties data for events or resources should always happen in the new resource_properties_data table which includes a longtext data column (and never in the legacy columns in resource or event). So, not sure how that stack trace could happen in Ocata.

I agree with therve that the change he linked to should fix the issue in Newton.

[Joe Talerico (jtaleric)]

This is Newton

[Steven Hardy (shardy)]

Ok thanks folks, apologies for the confusion re versions, it was indeed newton, so I cherry-picked the fix from therve ref https://review.openstack.org/#/c/443168/