keystone didn't start due to address already in use

[Russell Bryant]

http://logs.openstack.org/74/42474/13/gate/gate-tempest-devstack-vm-full/6640711/logs/screen-key.txt.gz

patch 13 of https://review.openstack.org/#/c/42474/

[Sean Dague (sdague)]

This looks like it's an issue where the devstack node may not have been clean when we started, possibly reused from another run, as otherwise there is nothing keystone should port conflict against.

[James E. Blair (corvus)]

According to both nodepool and jenkins logs, that node only ran one test. The output from the devstack-gate scripts seems consistent with it being the first run as well.

Perhaps we should add a netstat call to devstack-gate to help catch this in the future.

[Adam Spiers (adam.spiers)]

Jenkins just hit this again when verifying 49214:

https://review.openstack.org/#/c/49214/
http://logs.openstack.org/14/49214/1/check/check-swift-devstack-vm-functional/7509def/logs/screen-key.txt.gz

[Clark Boylan (cboylan)]

I think Jim's idea is a good one. Maybe we can trap the error then run `netstat -lnp` in devstack-gate. This should help us catch whatever process is listening on keystone's port.

[Matt Riedemann (mriedem)]

Just hit this again:

http://logs.openstack.org/63/60563/1/gate/gate-tempest-dsvm-full/066f11a/logs/screen-key.txt.gz

We don't have an e-r query for this, I'll look into adding one.