stack delete fails with Resource delete failed: UnicodeDecodeError

[Robert Pothier]

Getting the following on deleting a stack.

command used
heat create precise --template-file=InstanceGroup.template --parameters="ImageId=precise"

openstack@devstack-17:/opt/stack/heat$ git branch -v
* master 6f26ac6 Merge "Add keystone_authtoken section into the heat.conf.sample"

openstack@devstack-17:~/devstack$ heat list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| 82aa2108-23e0-42aa-8955-8a525dec7ebd | precise | DELETE_FAILED | 2013-10-16T22:12:30Z |

openstack@devstack-17:~/devstack$ heat stack-show precise
+----------------------+--------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2013-10-16T22:12:30Z |
| description | Template to create multiple instances. |
| disable_rollback | True |
| id | 82aa2108-23e0-42aa-8955-8a525dec7ebd |
| links | http://14.0.1.17:8004/v1/be7ab858f69648e3a929a58e4e3372c4/stacks/precise/82aa2108-23e0-42aa-8955-8a525dec7ebd |
| notification_topics | [] |
| parameters | { |
| | "NumInstances": "1", |
| | "SubnetUuid": "02b9d8f5-4f6f-4160-b99c-64160b16cc57", |
| | "AWS::StackName": "precise", |
| | "ImageId": "precise", |
| | "AWS::StackId": "arn:openstack:heat::be7ab858f69648e3a929a58e4e3372c4:stacks/precise/82aa2108-23e0-42aa-8955-8a525dec7ebd", |
| | "KeyName": "heat_key", |
| | "AWS::Region": "ap-southeast-1", |
| | "InstanceType": "m1.small" |
| | } |
| stack_name | precise |
| stack_status | DELETE_FAILED |
| stack_status_reason | Resource delete failed: UnicodeDecodeError: 'utf8' |
| | codec can't decode byte 0xdf in position 1: invalid |
| | continuation byte |
| template_description | Template to create multiple instances. |
| timeout_mins | 60 |
| updated_time | 2013-10-17T01:49:49Z |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------+

2013-10-17 16:24:07.846 18674 ERROR heat.engine.resource [-] Delete InstanceGroup "JobServerGroup"
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource Traceback (most recent call last):
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 527, in delete
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource handle_data = self.handle_delete()
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/autoscaling.py", line 217, in handle_delete
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource return self.delete_nested()
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/stack_resource.py", line 188, in delete_nested
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource delete_task.start()
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 136, in start
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource result = self._task(*self._args, **self._kwargs)
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/parser.py", line 571, in delete
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource user_creds = db_api.user_creds_get(stack.user_creds_id)
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/db/api.py", line 157, in user_creds_get
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource return IMPL.user_creds_get(context_id)
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/db/sqlalchemy/api.py", line 316, in user_creds_get
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource result['password'] = _decrypt(result['password'])
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource File "/opt/stack/heat/heat/db/sqlalchemy/api.py", line 136, in _decrypt
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource return unicode(value, 'utf-8')
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource UnicodeDecodeError: 'utf8' codec can't decode byte 0xdc in position 0: invalid continuation byte
2013-10-17 16:24:07.846 18674 TRACE heat.engine.resource

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "Template to create multiple instances.",

  "Parameters" : {
    "KeyName" : {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
      "Type" : "String",
      "Default" : "heat_key"
    },
    "InstanceType" : {
      "Description" : "Instance type",
      "Type" : "String",
      "Default" : "m1.small",
      "AllowedValues" : [ "m1.tiny", "m1.small", "m1.medium", "m1.large", "m1.xlarge" ],
      "ConstraintDescription" : "must be a valid EC2 instance type."
    },
    "ImageId" : {
      "Description" : "Name of the image to use",
      "Type" : "String",
      "Default" : "F17-x86_64-cfntools"
    },
    "NumInstances": {
      "Default": "1",
      "MinValue": "1",
      "MaxValue": "1001",
      "Description" : "Number of instances to create",
      "Type": "Number"
    },
    "SubnetUuid" : {
      "Description" : "Subnet UUID",
      "Default" : "02b9d8f5-4f6f-4160-b99c-64160b16cc57",
      "Type" : "String"
    }
  },
  "Resources" : {
    "JobServerGroup" : {
      "Type" : "OS::Heat::InstanceGroup",
      "Properties" : {
        "LaunchConfigurationName" : { "Ref" : "JobServerConfig" },
        "Size" : {"Ref": "NumInstances"},
        "AvailabilityZones" : { "Fn::GetAZs" : "" }
      }
    },

    "JobServerConfig" : {
      "Type" : "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId" : { "Ref" : "ImageId" },
        "InstanceType" : { "Ref" : "InstanceType" },
        "KeyName" : { "Ref" : "KeyName" },
        "NovaSchedulerHints": [ {"Key": "part", "Value": "long"},
                                {"Key": "ready", "Value": "short"} ],
        "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
          "#!/bin/bash -v\n"
        ]]}}
      }
    }
  }
}

[Robert Pothier (rpothier)]

This issue is seen occasionally when bringing up the above template and then deleting the template.
The are no errors on create.

[Steven Hardy (shardy)]

Sigh, I8n is hard :(

So the issue is we've added encoding to utf-8 in this patch https://review.openstack.org/#/c/46287/

But I'm guessing one of the users creating the stack has an accented character in their password, which is somehow getting to the DB API in some other encoding than utf8, e.g latin-1 (by virtue of your locale exported in your environment?)

E.g

>>> import sys
>>> print sys.stdout.encoding
UTF-8
>>> print unicode(u'\xe9'.encode('latin-1').encode('utf-8'), 'utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

One way to work around this is to use deferred_auth_method=trusts (in Havana Heat), then we don't store a password, but we need to figure out the best way to handle the encoding issue. atm I'm not sure how we do, if we don't know what encoding the string provided in the context is using.

[Steve Baker (steve-stevebaker)]

It would be good to confirm that your heat mysql database is using UTF-8 character set:

MariaDB [(none)]> use heat;
MariaDB [heat]> show variables like 'character_set_database';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+

MariaDB [heat]> show create table user_creds;
CREATE TABLE `user_creds` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  `username` varchar(255) DEFAULT NULL,
  `password` varchar(255) DEFAULT NULL,
  `tenant` varchar(1024) DEFAULT NULL,
  `auth_url` text,
  `tenant_id` varchar(256) DEFAULT NULL,
  `trustor_user_id` varchar(64) DEFAULT NULL,
  `trust_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8

[Robert Pothier (rpothier)]

mysql> use heat
Database changed
mysql> show variables like 'character_set_database';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+
1 row in set (0.00 sec)

mysql> show create table user_creds;
ERROR 1146 (42S02): Table 'heat.user_creds' doesn't exist
mysql>

[Robert Pothier (rpothier)]

mysql> show create table user_creds;
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| user_creds | CREATE TABLE `user_creds` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  `username` varchar(255) DEFAULT NULL,
  `password` varchar(255) DEFAULT NULL,
  `tenant` varchar(1024) DEFAULT NULL,
  `auth_url` text,
  `tenant_id` varchar(256) DEFAULT NULL,
  `trustor_user_id` varchar(64) DEFAULT NULL,
  `trust_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql>

[Steve Baker (steve-stevebaker)]

Please let me know if this bug is still valid.

[Robert Pothier (rpothier)]

I have not seen this bug recently.

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Steve Baker
> Sent: Tuesday, January 14, 2014 4:23 PM
> To: Rob Pothier (rpothier)
> Subject: [Bug 1241092] Re: stack delete fails with Resource delete failed:
> UnicodeDecodeError
>
> Please let me know if this bug is still valid.
>
> ** Changed in: heat
> Milestone: icehouse-2 => icehouse-3
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1241092
>
> Title:
> stack delete fails with Resource delete failed: UnicodeDecodeError
>
> Status in Orchestration API (Heat):
> Triaged
>
> Bug description:
> Getting the following on deleting a stack.
>
> command used
> heat create precise --template-file=InstanceGroup.template --
> parameters="ImageId=precise"
>
> openstack@devstack-17:/opt/stack/heat$ git branch -v
> * master 6f26ac6 Merge "Add keystone_authtoken section into the
> heat.conf.sample"
>
>
> openstack@devstack-17:~/devstack$ heat list
> +--------------------------------------+------------+-----------------+--------------------
> --+
> | id | stack_name | stack_status | creation_time |
> +--------------------------------------+------------+-----------------+--------------------
> --+
> | 82aa2108-23e0-42aa-8955-8a525dec7ebd | precise | DELETE_FAILED |
> 2013-10-16T22:12:30Z |
>
>
> openstack@devstack-17:~/devstack$ heat stack-show precise
> +----------------------+--------------------------------------------------------------------
> ------------------------------------------------------------+
> | Property | Value
> |
> +----------------------+--------------------------------------------------------------------
> ------------------------------------------------------------+
> | capabilities | []
> |
> | creation_time | 2013-10-16T22:12:30Z
> |
> | description | Template to create multiple instances.
> |
> | disable_rollback | True
> |
> | id | 82aa2108-23e0-42aa-8955-8a525dec7ebd
> |
> | links |
> http://14.0.1.17:8004/v1/be7ab858f69648e3a929a58e4e3372c4/stacks/precis
> e/82aa2108-23e0-42aa-8955-8a525dec7ebd |
> | notification_topics | []
> |
> | parameters | {
> |
> | | "NumInstances": "1",
> |
> | | "SubnetUuid": "02b9d8f5-4f6f-4160-b99c-64160b16cc57",
> |
> | | "AWS::StackName": "precise",
> |
> | | "ImageId": "precise",
> |
> | | "AWS::StackId":
> "arn:openstack:heat::be7ab858f69648e3a929a58e4e3372c4:stacks/precise/8
> 2aa2108-23e0-42aa-8955-8a525dec7ebd", |
> | | "KeyName": "heat_key",
> |
> | | "AWS::Region": "ap-southeast-1",
> |
> | | "InstanceType": "m1.small"
> |
> | | }
> |
> | stack_name ...

[Chris Friesen (cbf123)]

I'm currently seeing a bug like this with havana. Was there a known fix that went in for icehouse?

In my case though I can't see how it would be due to accented characters...it's a little private developer install with admin/admin/admin for username/password/tenant.

[Chris Friesen (cbf123)]

As a followup, in our case it's failing in heat/db/sqlalchemy/api.py", line 316, in user_creds_get(). The line looks like:

result['password'] = _decrypt(result['password'])

We're using postgres, and currently it looks like it's set to SQL_ASCII. Should we be changing to use utf-8 for heat? What about the other openstack databases?

[Steve Baker (steve-stevebaker)]

please change your postgres database to utf-8 and see if you can reproduce this issue.

[Steve Baker (steve-stevebaker)]

There looks like nothing actionable on this right now.