JJB KeyError in deep_format() during compare-xml-new

[Jeremy Stanley]

JJB's compare-xml-new test (when called from openstack-infra/config's run-layout.sh) seems to want to interpret the curly braces of a Bourne shell inline group code block within a YAML literal string as delimiting a parameter, and ends up erroring on the contents of that block while trying to perform a key lookup with it. It validates successfully if changed to use parentheses, but this will obviously run the function in a subshell and in some cases may change the intended result. We have similar constructs in production successfully, so this only seems to trip up JJB's parser when tested in the context of a configuration change.

It was tickled by change https://review.openstack.org/38176 introducing a job containing this builders list...

    builders:
      - devstack-inprogress
      - devstack-checkout
      - shell: |
          #!/bin/bash -xe
          export PYTHONUNBUFFERED=true
          export DEVSTACK_GATE_TEMPEST=0
          export DEVSTACK_GATE_EXERCISES=0
          if [ "{mirror}" == "no-mirror" ]; then
              MIRROR="--no-mirror"
          fi
          function gate_hook {
              bash $GATE_SCRIPT_DIR/devstack-vm-gate.sh && \
              bash -xe $BASE/new/pbr/tools/integration.sh $MIRROR $PROJECTS
          }
          export -f gate_hook
          cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
          ./safe-devstack-vm-gate-wrap.sh
      - link-logs

The exception thrown during the gate-config-layout test was...

    compare-xml-new runtests: commands[0] | jenkins-jobs test -o .test/new/out/ .test/new/config/
    Traceback (most recent call last):
      File ".tox/compare-xml-new/bin/jenkins-jobs", line 9, in <module>
        load_entry_point('jenkins-job-builder==0.0.3.c9695f1', 'console_scripts', 'jenkins-jobs')()
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/cmd.py", line 102, in main
        output_dir=options.output_dir)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 424, in update_job
        parser.generateXML(names)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 156, in generateXML
        self.getXMLForTemplateJob(d, template, jobs_filter)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 171, in getXMLForTemplateJob
        expanded = deep_format(template, params)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 51, in deep_format
        ret[item] = deep_format(obj[item], paramdict)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 47, in deep_format
        ret.append(deep_format(item, paramdict))
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 51, in deep_format
        ret[item] = deep_format(obj[item], paramdict)
      File "/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/local/lib/python2.7/site-packages/jenkins_jobs/builder.py", line 43, in deep_format
        ret = obj.format(**paramdict)
    KeyError: '\n bash $GATE_SCRIPT_DIR/devstack-vm-gate'
    ERROR: InvocationError: '/home/jenkins/workspace/gate-config-layout/.test/jenkins-job-builder/.tox/compare-xml-new/bin/jenkins-jobs test -o .test/new/out/ .test/new/config/'

Full log: http://logs.openstack.org/76/38176/4/check/gate-config-layout/e6f5add/console.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/40973

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/40973
Committed: http://github.com/openstack-infra/jenkins-job-builder/commit/ca509654c30ddfb2f2a6617a9de1e87f7d87a8cc
Submitter: Jenkins
Branch: master

commit ca509654c30ddfb2f2a6617a9de1e87f7d87a8cc
Author: Jeremy Stanley <email address hidden>
Date: Tue Aug 13 03:27:34 2013 +0000

    Document escaping braces in template literals

    * doc/source/configuration.rst: In a literal string within a
    template, deep_format will interpret braces as delimiting an
    expansion parameter unless doubled up. Document this.

    Change-Id: I703f03f89436a09d64b79b147bec6441e1f8cc38
    Closes-Bug: #1210293