multi-proc devstack sometimes fails on encryption key

[Eric K]

Occasionally when starting multi-process Congress using devstack, the following error occurs. Gate is occasionally affected as a result.

Started Devstack <email address hidden>.
Traceback (most recent call last):
  File "/usr/local/bin/congress-server", line 10, in <module>
    sys.exit(main())
  File "/opt/stack/new/congress/congress/server/congress_server.py", line 148, in main
    encryption.initialize_key()
  File "/opt/stack/new/congress/congress/encryption.py", line 73, in initialize_key
    __fernet = Fernet(__key)
  File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 37, in __init__
    "Fernet key must be 32 url-safe base64-encoded bytes."
ValueError: Fernet key must be 32 url-safe base64-encoded bytes.

The cause appears to be that the second Congress node reads the key file before the first Congress node finishes writing it.
https://github.com/openstack/congress/blob/master/devstack/plugin.sh#L206

Some possible fixes:
- change the devstack plugin script doing the launch
- change the encryption key initialization code to wait until a valid key is written to file

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/495989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/495989
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=d014b918367eda4f807303eb449e7ec0141d6711
Submitter: Jenkins
Branch: master

commit d014b918367eda4f807303eb449e7ec0141d6711
Author: Eric Kao <email address hidden>
Date: Mon Aug 21 14:47:03 2017 -0700

    use encryption key only on datasource node

    Not needed on other nodes. Also occassionally causes
    devstack start-up issue.

    Closes-Bug: 1712177

    Change-Id: I2f359f503d5ef02ac26555b0edb75f67cc89a459

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/congress 7.0.0.0b1 development milestone.