parameter_defaults breaks deployment json

[Jiří Stránský]

Failed jobs with similar but not the same errors:

[16:51:28] <jistr> [INFO] Error: 757: unexpected token at 'EATE"}' at /var/lib/heat-config/heat-config-puppet/1a109404-8d0f-45b0-83c4-8fb36f2337f6.pp:16 on node ov-y2fisc7wl3s-0-bgus47tixil2-controller-lpqz2dy77wih.novalocal
[16:51:49] <jistr> [INFO] Error: 757: unexpected token at '"}' at /var/lib/heat-config/heat-config-puppet/b1dbc213-93ac-4080-b054-e971d80ae0c6.pp:16 on node ov-bhxoilynjag-0-nd6ojit5llhr-controller-ztuoo4iluxw5.novalocal
[16:51:58] <jistr> [INFO] Error: 757: unexpected token at 'CREATE"}' at /var/lib/heat-config/heat-config-puppet/17d86b9b-4628-4a0d-b110-99acf1eae962.pp:16 on node ov-w7m2elw72n5-0-gyulzvxskih2-controller-75ddyyqw2kj5.novalocal

[16:52:51] <jistr> http://logs.openstack.org/11/163211/3/check-tripleo/check-tripleo-ironic-overcloud-f20puppet-nonha/1c2cfb7/
[16:52:55] <jistr> http://logs.openstack.org/44/164144/4/check-tripleo/check-tripleo-ironic-overcloud-f20puppet-nonha/eb94373/
[16:53:07] <jistr> http://logs.openstack.org/13/165813/1/check-tripleo/check-tripleo-ironic-overcloud-f20puppet-nonha/09408ab/

/etc/puppet/hieradata/heat_config_overcloud-ControllerNodesPostDeployment-cn5uvlkz4xfg-ControllerPuppetConfig-nj5bruyo7jm6.json generates misformatted:

{"deploy_resource_name": "0", "deploy_signal_id": "http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A78088effc0934acd900dabccbd8ed9c2%3Astacks%2Fovercloud-ControllerNodesPostDeployment-cn5uvlkz4xfg-ControllerDeploymentServicesBase_Step2-3jdzckpwxbme%2F8c7d4bd3-b295-4ebb-a09e-99bf0c6a61f3%2Fresources%2F0?Timestamp=2015-03-19T16%3A31%3A28Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=a42c94d713124048973f03317e71bc29&SignatureVersion=2&Signature=iYVjYXwGVMHERRRPL5ODsZR3rTeZQnpAVhPbQ4i5K%2BM%3D", "step": "2", "deploy_server_id": "1412c36d-fdd2-4317-b562-a7deb2488144", "deploy_stack_id": "overcloud-ControllerNodesPostDeployment-cn5uvlkz4xfg-ControllerDeploymentServicesBase_Step2-3jdzckpwxbme/8c7d4bd3-b295-4ebb-a09e-99bf0c6a61f3", "deploy_action": "CREATE"}"}

Note this part, which breaks the file and makes it an invalid JSON:

"deploy_action": "CREATE"}"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/165986

[Dan Prince (dan-prince)]

Looks like reverting the parameter_defaults changes fixes it for me. No idea how this passed CI, and now we are seeing failures....

https://review.openstack.org/165986

[Steven Hardy (shardy)]

I'm not clear how the parameter_defaults for EnablePackageInstall can break the json output for the deployment, but it seems like it could be a heat bug - adding heat to the bug so we can investigate and determine if that is the case.

Does anyone have any more minimal way of reproducing this, or have any additional analysis of the connection between the parameter_defaults nad the broken deployment?

[Jiří Stránský (jistr)]

Adding more info: reverting the parameter_defaults doesn't fix the issue for me (i still reproduced it after commenting out the parameter_defaults bit). It might make the issue less likely to occur (?).

Also, as hinted by the 3 different error messages in the bug description, the generated strings seem to differ from case to case, in my today's deployment attempts i had this bit in the file:

"CREATE"}"}

and then this:

"CREATE"}EATE"}

and then on 3rd try the deployment succeeded. All the deployments were made without parameter_defaults in place.

Seems like it might be some kind of race condition? :-/

[Angus Salkeld (asalkeld)]

Just set the heat part of this to incomplete as someone needs to work this up a bit more.
Not much to go on so far.

[Steven Hardy (shardy)]

@asalkeld: Thanks, I'm planning to do some testing on this, just added heat so we'd follow up on why the revert is needed, I'll assign it to myself (although I probably can't work on this until next week, so feel free if anyone has time to investigate before then)

[Jiří Stránský (jistr)]

Gerrit didn't update the bug so i'm updating manually. It looks like the problem is in the heat-config-puppet image element and i believe this patch fixes it (i've had 5 of 5 successful deployments with it now), see the commit message for details:

https://review.openstack.org/#/c/168328/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/165986
Reason: This did seem to decrease the likelihood of the JSON issue but it isn't the correct fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168328
Committed: https://git.openstack.org/cgit/openstack/heat-templates/commit/?id=f00cf8fbb89e04851d9cb0c5f2f99c80beb7d5c3
Submitter: Jenkins
Branch: master

commit f00cf8fbb89e04851d9cb0c5f2f99c80beb7d5c3
Author: Jiri Stransky <email address hidden>
Date: Fri Mar 27 13:31:10 2015 +0100

    Fix hiera deployment JSON generation

    We were hitting strange intermittent errors in TripleO where the hiera
    JSON file generated by Heat was not even valid JSON, ending with bits of
    text which seem out of place, and of varying length, e.g.

    "CREATE"}"}

    or

    "CREATE"}EATE"}

    The cause could be that the generation of the file runs multiple times,
    and if the first generated JSON is longer than the subsequent one, the
    ending of the previous JSON is still present in the file, because the
    generation opens the file with O_CREAT (create if not exists) and
    O_WRONLY (write only) flags, but not with O_TRUNC (truncate file before
    writing). This patch adds the truncation to fix the issue.

    Change-Id: Icf184f973decec2eb7de1dee9959b60774d83eb6
    Closes-Bug: #1434187