stack-update adding a parameter leads to heat stack-list breaking

Bug #1354962 reported by Robert Collins on 2014-08-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Zane Bitter

Bug Description

I did a stack-update of a stack that:
 - changed format (to HOT)
 - added a parameter ('Debug')
 - supplied that parameter in the new environment file

and now CFN metadata polling is broken, and heat stack-list errors as well.

In the API log I see:
2014-08-10 23:20:38.890 6176 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET /v1/43a9170dc5cb47b88ffd40acff0bf0e9/stacks Accept: application/json process_request /opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:51
2014-08-10 23:20:38.890 6176 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:66
2014-08-10 23:20:38.891 6176 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /opt/stack/venvs/heat/local/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:666
2014-08-10 23:20:38.891 6176 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-
Role _remove_auth_headers /opt/stack/venvs/heat/local/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:725
2014-08-10 23:20:38.893 6176 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-08-10 23:20:38.936 6176 DEBUG requests.packages.urllib3.connectionpool [-] "GET /v2.0/tokens/revoked HTTP/1.1" 200 698 _make_request /opt/stack/venvs/heat/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:350
2014-08-10 23:20:38.964 6176 DEBUG iso8601.iso8601 [-] Parsed 2014-08-11T03:20:38Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'03', 'daydash': u'11', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'38', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'08', 'day': None, 'minute': u'20'} with default timezone <iso8601.iso8601.Utc object at 0x7f9012c64c90> parse_date /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:184
2014-08-10 23:20:38.965 6176 DEBUG iso8601.iso8601 [-] Got u'2014' for 'year' with default None to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got u'08' for 'monthdash' with default 1 to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got 8 for 'month' with default 8 to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got u'11' for 'daydash' with default 1 to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got 11 for 'day' with default 11 to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got u'03' for 'hour' with default None to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.966 6176 DEBUG iso8601.iso8601 [-] Got u'20' for 'minute' with default None to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.967 6176 DEBUG iso8601.iso8601 [-] Got u'38' for 'second' with default None to_int /opt/stack/venvs/heat/local/lib/python2.7/site-packages/iso8601/iso8601.py:140
2014-08-10 23:20:38.967 6176 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache _cache_put /opt/stack/venvs/heat/local/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1234
2014-08-10 23:20:38.968 6176 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: de92d8d91cdc4ed79d0819caa7596d3d with project_id : 43a9170dc5cb47b88ffd40acff0bf0e9 and roles: _member_,admin _build_user_headers /opt/stack/venvs/heat/local/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1021
2014-08-10 23:20:38.971 6176 DEBUG heat.openstack.common.policy [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 None] Rules successfully reloaded load_rules /opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/openstack/common/policy.py:242
2014-08-10 23:20:38.972 6176 DEBUG routes.middleware [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] Matched GET /43a9170dc5cb47b88ffd40acff0bf0e9/stacks __call__ /opt/stack/venvs/heat/local/lib/python2.7/site-packages/routes/middleware.py:100
2014-08-10 23:20:38.972 6176 DEBUG routes.middleware [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] Route path: '/{tenant_id}/stacks', defaults: {'action': u'index', 'controller': <heat.common.wsgi.Resource object at 0x7f901155a650>} __call__ /opt/stack/venvs/heat/local/lib/python2.7/site-packages/routes/middleware.py:102
2014-08-10 23:20:38.972 6176 DEBUG routes.middleware [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] Match dict: {'action': u'index', 'tenant_id': u'43a9170dc5cb47b88ffd40acff0bf0e9', 'controller': <heat.common.wsgi.Resource object at 0x7f901155a650>} __call__ /opt/stack/venvs/heat/local/lib/python2.7/site-packages/routes/middleware.py:103
2014-08-10 23:20:38.973 6176 DEBUG oslo.messaging._drivers.amqpdriver [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] MSG_ID is 08c47b6d71ba4d8cadb4519b1d23ce9c _send /opt/stack/venvs/heat/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:378
2014-08-10 23:20:38.974 6176 DEBUG oslo.messaging._drivers.amqp [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] UNIQUE_ID is 4986dc05f78940c886b2b3e445878f38. _add_unique_id /opt/stack/venvs/heat/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338
2014-08-10 23:20:38.976 6176 DEBUG amqp [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] Closed channel #1 _do_close /opt/stack/venvs/heat/local/lib/python2.7/site-packages/amqp/channel.py:104
2014-08-10 23:20:38.976 6176 DEBUG amqp [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] using channel_id: 1 __init__ /opt/stack/venvs/heat/local/lib/python2.7/site-packages/amqp/channel.py:80
2014-08-10 23:20:38.977 6176 DEBUG amqp [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 ] Channel open _open_ok /opt/stack/venvs/heat/local/lib/python2.7/site-packages/amqp/channel.py:438
2014-08-10 23:20:39.011 6176 DEBUG heat.common.serializers [req-e0fa3f3f-b3cf-4c67-bdc9-9017e7729603 None] JSON response : {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "The Parameter (Debug) was not defined in template.", "traceback": "Traceback (most recent call last):\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/service.py\", line 63, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/service.py\", line 458, in list_stacks\n return [api.format_stack(stack) for stack in stacks]\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/parser.py\", line 233, in load_all\n yield cls._from_db(context, stack, resolve_data=resolve_data)\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/parser.py\", line 250, in _from_db\n use_stored_context=use_stored_context)\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/parser.py\", line 114, in __init__\n context=self.context)\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/parameters.py\", line 437, in validate\n self._validate_user_parameters()\n\n File \"/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/parameters.py\", line 480, in _validate_user_parameters\n raise exception.UnknownUserParameter(key=param)\n\nUnknownUserParameter: The Parameter (Debug) was not defined in template.\n", "type": "UnknownUserParameter"}, "title": "Bad Request"} to_json /opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/common/serializers.py:43

Zane Bitter (zaneb) wrote :

I believe this patch, which I posted on Friday, will resolve the issue: https://review.openstack.org/#/c/112936/

However, it's buried in the middle of a long queue. If this is urgent I can move it to the front (I don't think it actually depends on any of the earlier stuff).

Changed in heat:
assignee: nobody → Zane Bitter (zaneb)
status: New → Confirmed
milestone: none → juno-3
status: Confirmed → In Progress
Zane Bitter (zaneb) wrote :

BTW I assume the stack update failed? That should be the only way to get into this state, so if not that would be worthy of immediate investigation... of course I realise it's impossible to tell through the API because of this bug.

Reviewed: https://review.openstack.org/112936
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=acf39978ad91bb6d49cb23459f0c521f3262afac
Submitter: Jenkins
Branch: master

commit acf39978ad91bb6d49cb23459f0c521f3262afac
Author: Zane Bitter <email address hidden>
Date: Mon Aug 11 17:26:14 2014 -0400

    Don't validate parameters when loading a stack

    All kinds of Frankentemplates can now be stored in the database. We don't
    want to refuse to load them again. Only validate the parameters when we
    actually validate the stack.

    Change-Id: I8bff116f1a9c16be2bd10a8da685ef9571e1101f
    Closes-bug: #1354962

Changed in heat:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-09-05
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-10-16
Changed in heat:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers