cfg validator agent fails to start TypeError: unorderable types: str() < NoneType()

[Eric K]

Likely because of changes introduced by oslo.config 5.2.0, congress-cfg-validator-agt now fails to start in py35 devstack gate with the following error. No problem in py27 dsvm.

-- Logs begin at Thu 2018-01-11 21:27:03 UTC, end at Thu 2018-01-11 21:58:31 UTC. --
Jan 11 21:58:08.537376 ubuntu-xenial-inap-mtl01-0001828423 systemd[1]: Started Devstack <email address hidden>.
Jan 11 21:58:09.996044 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 CRITICAL congress [-] Unhandled error: TypeError: unorderable types: str() < NoneType()
Jan 11 21:58:09.996260 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress Traceback (most recent call last):
Jan 11 21:58:09.996423 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/usr/local/bin/congress-cfg-validator-agt", line 10, in <module>
Jan 11 21:58:09.996683 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress sys.exit(main())
Jan 11 21:58:09.996852 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 427, in main
Jan 11 21:58:09.997014 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress agent = ValidatorAgentEndpoint()
Jan 11 21:58:09.997197 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 348, in __init__
Jan 11 21:58:09.997356 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress self.config_manager = ConfigManager(self.host, service_files)
Jan 11 21:58:09.997510 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 263, in __init__
Jan 11 21:58:09.997662 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress self.register_service(service_name, files)
Jan 11 21:58:09.997813 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 317, in register_service
Jan 11 21:58:09.997973 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress self.register_config(config_path, template_path, service_name)
Jan 11 21:58:09.998125 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 296, in register_config
Jan 11 21:58:09.998276 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress or self.add_template(template_path))
Jan 11 21:58:09.998430 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 280, in add_template
Jan 11 21:58:09.998588 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress template = Template.load(template_path)
Jan 11 21:58:09.998762 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 221, in load
Jan 11 21:58:09.998914 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress template.namespaces.append(Namespace.load(namespace))
Jan 11 21:58:09.999077 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/agent.py", line 157, in load
Jan 11 21:58:09.999227 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress json_data = validator_generator.generate_ns_data(name)
Jan 11 21:58:09.999379 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/opt/stack/new/congress/congress/cfg_validator/agent/generator.py", line 123, in generate_ns_data
Jan 11 21:58:09.999530 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress return OptionJsonEncoder(sort_keys=True).encode(groups)
Jan 11 21:58:09.999687 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
Jan 11 21:58:09.999839 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress chunks = self.iterencode(o, _one_shot=True)
Jan 11 21:58:09.999997 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
Jan 11 21:58:10.000155 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress return _iterencode(o, 0)
Jan 11 21:58:10.000307 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress TypeError: unorderable types: str() < NoneType()
Jan 11 21:58:10.000458 ubuntu-xenial-inap-mtl01-0001828423 congress-cfg-validator-agt[4771]: 2018-01-11 21:58:09.992 4771 ERROR congress
Jan 11 21:58:10.186745 ubuntu-xenial-inap-mtl01-0001828423 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Jan 11 21:58:10.186951 ubuntu-xenial-inap-mtl01-0001828423 systemd[1]: <email address hidden>: Unit entered failed state.
Jan 11 21:58:10.186981 ubuntu-xenial-inap-mtl01-0001828423 systemd[1]: <email address hidden>: Failed with result 'exit-code'.

http://logs.openstack.org/93/532593/2/check/congress-devstack-py35-api-mysql/630e80f/logs/screen-congress-agent.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532593
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=d86f680f09a7660737b6a53794e1bc8076a40d69
Submitter: Zuul
Branch: master

commit d86f680f09a7660737b6a53794e1bc8076a40d69
Author: Eric K <email address hidden>
Date: Wed Jan 10 10:29:28 2018 -0800

    Update cfgvalidator_driver for oslo.config 5.2

    oslo.config 5.2 begins to use a different representation of
    choices (OrderedDict). We convert back to simple list to
    have consistent output regardless of oslo.config version

    unit tests have been failing since the change in upper-constraint
    to oslo.config 5.2

    Also temporarily disabled cfgvalidator in dsvm gate
    while a blocking issue is being resolved, so that critical
    queens-3 patches can go in asap.

    Related-Bug: #1742833

    Change-Id: Ibec47e398686793258eb4c8d9fb056e51cdc1c49

[Pierre Crégut (pcregut)]

Here is the likely culprit:

from json import *
from collections import *
JSONEncoder(sort_keys=True).encode(OrderedDict([(None,'a'),('a','b')]))

Python3 will fail, Python2.7 will use the string "null" instead. I guess there is an option with
None as a valid value.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533248
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=724810874560da880da696d56f2451ff504f64fc
Submitter: Zuul
Branch: master

commit 724810874560da880da696d56f2451ff504f64fc
Author: Pierre Crégut <email address hidden>
Date: Fri Jan 26 11:10:58 2018 +0100

    Fix serialization of choices and reenable congress-agent.

    oslo_config uses OrderedDict for choices in
    options types. This serialize it as list of tuples.
    Order is preserved and types are naturally
    deserialized back from those tuples.

    Congress-agent is reenabled. It can be disabled
    in devstack on the value of ENABLE_CONGRESS_AGENT
    (default is True).

    Driver will initialize table to empty at first poll not
    waiting for an initial response from agents.

    Closes-Bug: #1742833
    Change-Id: If1423e824fc457bab89f4bcd12be8826c50f5f69

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/congress 7.0.0.0rc1 release candidate.