CORS functional tests sometimes fail with `invalid argument: can't kill an exited process`

[Tim Burke]

6 or so times in the past week, the CORS func tests failed becuase all tests were skipped. Most drivers are expected to skip, but Firefox should have been set up properly; instead, it reports reports

   ok 1 - firefox # skip Message: invalid argument: can't kill an exited process

Not sure what that means or where it's bubbling up, but the improved traceback logging in https://review.opendev.org/c/openstack/swift/+/780129 should help the next time we see it.

[Tim Burke (1-tim-z)]

FWIW, they all report installing Mozilla Firefox 78.8.0esr, but there are passing builds that use that same version. Similarly, we've seen the same version of selenium on both passing and failing jobs (with FF 78.8) -- I don't think it comes down to a version conflict or something.

Will report a traceback once I've got one.

[Tim Burke (1-tim-z)]

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/swift/test/cors/main.py", line 182, in run
    browser = driver()
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
    keep_alive=True)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
WebDriverException: Message: invalid argument: can't kill an exited process

Looks like I'll be digging into selenium...

[Tim Burke (1-tim-z)]

OK, error message seems to be coming out of the geckodriver, not selenium exactly -- and the log *there* has

   Error: cannot open display: :99

when the job fails, but not when it passes. So I guess it's maybe one of xvfb-run, firefox, or geckodriver?

[Alistair Coles (alistair-coles)]

I've seen this failure mode a few times, stumbled across https://stackoverflow.com/questions/52534658/webdriverexception-message-invalid-argument-cant-kill-an-exited-process-with

So trying https://review.opendev.org/c/openstack/swift/+/781084

[Tim Burke (1-tim-z)]

I think it's fair to call this closed -- I haven't seen this particular failure mode for a while, likely ever since https://review.opendev.org/c/openstack/swift/+/781084 landed.