zuuls fails to reload/stop due to missing PID

[Ramy Asselin]

I've noticed zuul failing to reload the configuration and not able to stop due to a missing PID.

I investigated and root caused the issue.

Steps to reproduce:
1. Create an invalid layout.yaml file. In my case I had (abbreviated to relevant lines). Notice the "failure-message" property value on line 7 is long and doesn't have quotes:

1 includes:
2 - python-file: openstack_functions.py
3
4 pipelines:
5 - name: check
6 description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
7 failure-message: Build failed. For information on how to proceed, see https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures. For 3rd party ci contact info: https://wiki.openstack.org/wiki/ThirdPartySystems/HP_Storage_CI
    manager: IndependentPipelineManager

2. When you install zuul this via puppet, the badly formatted file is attempted to be used by zuul. Zuul will keep running, but not the previous configuration.
3. sudo service zuul start
will succeed, the pid will exist for a few seconds, and then disappear.
zuul-server process will keep running.
4. sudo service zuul stop will fail due to missing pid.

Running manually using -d option shows the stack trace:

zuul@ci:~$ zuul-server -d
Traceback (most recent call last):
  File "/usr/local/bin/zuul-server", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/zuul/cmd/server.py", line 243, in main
    server.main()
  File "/usr/local/lib/python2.7/dist-packages/zuul/cmd/server.py", line 197, in main
    self.sched.reconfigure(self.config)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 534, in reconfigure
    event.wait()
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 835, in process_management_queue
    self._doReconfigureEvent(event)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 622, in _doReconfigureEvent
    self.config.get('zuul', 'layout_config'))
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 219, in _parseConfig
    data = yaml.load(config_file)
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 220, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 580, in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "/etc/zuul/layout.yaml", line 7, column 169

Sure enough, adding quotes to line 7

Expected result is that this stack trace should be included in the zuul-debug.log file to aid in debugging and resolving.

[Ramy Asselin (ramy-asselin)]

Another workaround is to verify the configuration first before attempting to reload.
e.g. /usr/local/bin/zuul-server -t

diff showing a puppet script changed to verify before reloading:
https://github.com/rasselin/os-ext-testing/commit/bdc879da182b20ce6868cf6ad8890af0d7c74739