A bad or out-of-range value was supplied:The Resource (0) could not be found in Stack

[Jiří Stránský]

When reporting a StructuredDeployments completion from os-collect-config to Heat, it seems like some of the resources "disappeared" from Heat. This is the relevant bit from os-collect-config log:

říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + rm /tmp/tmp.kir9ukmL2y
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + '[' 200 '!=' 200 ']'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + for dep in '${DEPLOYMENTS}'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ echo 'b55bd7e3-a653-4356-8929-3db64ba7debehttp://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ sed 's/http.*$//'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + DEPLOY_ID=b55bd7e3-a653-4356-8929-3db64ba7debe
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ sed 's/^.*http/http/'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ echo 'b55bd7e3-a653-4356-8929-3db64ba7debehttp://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + DEPLOY_URL='http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + echo 'Signalling os-apply-config deployment b55bd7e3-a653-4356-8929-3db64ba7debe http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: Signalling os-apply-config deployment b55bd7e3-a653-4356-8929-3db64ba7debe http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + call_curl_deployment POST 'http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D' 'os-apply-config deployment b55bd7e3-a653-4356-8929-3db64ba7debe completed'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + local method=POST
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + local 'url=http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + local 'stdout=os-apply-config deployment b55bd7e3-a653-4356-8929-3db64ba7debe completed'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ mktemp
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + local output=/tmp/tmp.keF9NkpGyi
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ++ curl -s -w '%{http_code}' -X POST -H Content-Type: -o /tmp/tmp.keF9NkpGyi --data-binary '{"deploy_stdout": "os-apply-config deployment b55bd7e3-a653-4356-8929-3db64ba7debe completed", "deploy_status_code": "0"}' 'http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Ac461be5bba4f4575abcc86610de1ae44%3Astacks%2Fovercloud-VipDeployment-fjw47zvc4ryu%2F0746823e-5e62-47bb-8dac-8b67d9abbc59%2Fresources%2F0?Timestamp=2015-10-07T12%3A49%3A13Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=739f4ef4348046b89c3af92a2670c7b0&SignatureVersion=2&Signature=f4LcIXheWjdZj1Nda%2FDWdifDQLiM8xkVqgOzGni2Nmo%3D'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + status=400
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + cat /tmp/tmp.keF9NkpGyi
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: <ErrorResponse><Error><Message>A bad or out-of-range value was supplied:The Resource (0) could not be found in Stack overcloud-VipDeployment-fjw47zvc4ryu.
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: Traceback (most recent call last):
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 305, in wrapped
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: return func(self, ctx, *args, **kwargs)
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1385, in resource_signal
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: self._verify_stack_resource(stack, resource_name)
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1328, in _verify_stack_resource
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: stack_name=stack.name)
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: ResourceNotFound: The Resource (0) could not be found in Stack overcloud-VipDeployment-fjw47zvc4ryu.
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: </Message><Code>InvalidParameterValue</Code><Type>Sender</Type></Error></ErrorResponse>+ rm /tmp/tmp.keF9NkpGyi
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + '[' 400 '!=' 200 ']'
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: + exit 1
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: [2015-10-07 12:49:56,405] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1]
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: [2015-10-07 12:49:56,405] (os-refresh-config) [ERROR] Aborting...
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:49:56.415 1090 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1
říj 07 12:49:56 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:49:56.416 1090 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.
říj 07 12:50:28 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:50:28.348 1090 WARNING os-collect-config [-] Source [request] Unavailable.
říj 07 12:50:28 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:50:28.348 1090 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping
říj 07 12:50:28 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:50:28.348 1090 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data'])
říj 07 12:50:28 overcloud-controller-0 os-collect-config[1090]: 2015-10-07 12:50:28.349 1090 WARNING os_collect_config.zaqar [-] No auth_url configured.
říj 07 12:50:28 overcloud-controller-0 os-collect-config[1090]: [2015-10-07 12:50:28,395] (os-refresh-config) [INFO] Starting phase configure

[Jiří Stránský (jistr)]

Heat version:

[stack@instack ~]$ rpm -q openstack-heat-api openstack-heat-engine openstack-heat-api-cfn
openstack-heat-api-6.0.0-dev44.el7.centos.noarch
openstack-heat-engine-6.0.0-dev44.el7.centos.noarch
openstack-heat-api-cfn-6.0.0-dev44.el7.centos.noarch

Template of the problematic stack:

https://github.com/openstack/tripleo-heat-templates/blob/a328c134b1776bd34b720171967af5c94014529f/overcloud-without-mergepy.yaml#L1137-L1164

The TripleO deployment had 3 controllers, so in the 'servers' parameter for VipDeployment there were 3 servers.

[Steven Hardy (shardy)]

Has there been any reoccurance of this issue? I've not managed to reproduce it at all, so I may have to mark this incomplete unless we can figure out a reproducer.

[Jiří Stránský (jistr)]

I remember i've seen it several times back when i reported it, but i haven't seen it pop up lately.

[Steven Hardy (shardy)]

Per comment #6 I'm closing this invalid - please reopen if it occurs again and we can investigate further

[Launchpad Janitor (janitor)]

[Expired for heat because there has been no activity for 60 days.]