autopkgtest failures: Chromium-Related in Tests

[Thomas Ward]

Hello.

Currently, the kopano-webapp autopkgtests are failing. Three of these are related to Chromium crashes/failures (possibly because they're snapped) within the Selenium calls.

The fourth is a snap related issue in general.

These autopkgtest failures are blocking HTTPD packages in Proposed (nginx, apache, etc.)

------
Relevant Links:
amd64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/amd64/k/kopano-webapp/20190624_082152_0ef6d@/log.gz

arm64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/arm64/k/kopano-webapp/20190624_084645_0ef6d@/log.gz

armhf: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/armhf/k/kopano-webapp/20190624_082623_0ef6d@/log.gz

i386: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan/eoan/i386/k/kopano-webapp/20190624_082523_0ef6d@/log.gz

[Iain Lane (laney)]

all except armhf look like a problem with the Chromium snap

armhf will probably become that same problem once we're able to run snaps there (need to look into what's required)

[Thomas Ward (teward)]

Iain:

Sounds like armhf needs to be badtested then for kopano-webapp, or at least that failure ignored for now. THat one I can ask be ignored. The rest of them sounds like a Chromium snap issue, indeed.

[Olivier Tilloy (osomon)]

I'm unable to reproduce the test failure in an amd64 eoan VM. The smoke tests consistently pass there. Sounds like this might be something specific to the autopkgtest infrastructure. I'm going to try and reproduce in a similar test environment.

[Olivier Tilloy (osomon)]

I can now reproduce the failure by running the test with nosetests3 manually.

Removing the "--log-path=/tmp/chromedriver.log" service arg fixes the issue. The confined chromedriver bails out when it fails to write there:

  $ /snap/bin/chromium.chromedriver --log-path=/tmp/chromedriver.log
  Starting ChromeDriver 75.0.3770.90 (a6dcaf7e3ec6f70a194cc25e8149475c6590e025-refs/branch-heads/3770@{#1003}) on port 9515
  Only local connections are allowed.
  Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
  Failed to redirect stderr to log file.
  Unable to initialize logging. Exiting...

Given that the log file isn't exposed as a retrievable artefact when autopkgtests are run, I suggest we just remove it.
If it turns out that a significant number of existing selenium tests do the same, we could consider patching chromedriver in the snap to ignore that argument, instead of bailing out.

[Olivier Tilloy (osomon)]

Submitted suggested change to the debian package: https://salsa.debian.org/giraffe-team/kopano-webapp/merge_requests/1

[Iain Lane (laney)]

Still fails for me with the patch I'm afraid, sorry (autopkgtest w/qemu runner)

test_login (test_webapp.TestWebApp) ... ERROR

======================================================================
ERROR: test_login (test_webapp.TestWebApp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.yhOtVy/build.wUU/src/debian/tests/test_webapp.py", line 29, in setUp
    service_args=['--verbose'])
  File "/usr/lib/python3/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed
  (unknown error: unable to discover open pages)
  (The process started from chrome location /snap/chromium/772/usr/lib/chromium-browser/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

-------------------- >> begin captured logging << --------------------
selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:35567/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"extensions": [], "args": ["--headless", "--no-sandbox", "--window-size=1200x900"]}}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"extensions": [], "args": ["--headless", "--no-sandbox", "--window-size=1200x900"]}}}
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:35567
urllib3.connectionpool: DEBUG: http://127.0.0.1:35567 "POST /session HTTP/1.1" 500 952
selenium.webdriver.remote.remote_connection: DEBUG: Finished Request
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 5.848s

FAILED (errors=1)
autopkgtest [17:05:32]: test smoke: -----------------------]
autopkgtest [17:05:33]: test smoke: - - - - - - - - - - results - - - - - - - - - -
smoke FAIL non-zero exit status 1

[Iain Lane (laney)]

Uploaded it to my PPA and ran it on the real environment (to dispel any doubt that it's something local)

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan-laney-ppa/eoan/amd64/k/kopano-webapp/20190628_164752_2b134@/log.gz

same

[Olivier Tilloy (osomon)]

And I can reproduce the test failure in a qemu VM.

Interesting observations: running the chromium snap once ("snap run chromium --headless" or "chromium --version" or "chromium-browser --version") after installation and before running the tests make them pass. Deleting ~/snap/chromium after that makes the tests fail again.
Alternatively, specifying chrome_options.binary_location="/snap/chromium/current/command-chromium.wrapper" also makes the tests pass, without a prior run of the chromium snap.

[Olivier Tilloy (osomon)]

I'm building a chromium snap with a patched chromedriver that defaults to "/snap/chromium/current/command-chromium.wrapper" for the binary_location option. Once done I'll test and confirm whether this fixes the kopano-webapp autopkgtests without changes.

[Olivier Tilloy (osomon)]

I can confirm that the patched chromedriver fixes the kopano-webapp autopkgtests.
It is now building in the candidate channel for all supported architectures, I'll mark the bug fixed once it hits the stable channel.

[Olivier Tilloy (osomon)]

Fixed for amd64.
Builds for i386, armhf and arm64 are still going.

[Olivier Tilloy (osomon)]

Fixed for i386 and armhf.
arm64 builds are failing for an unrelated reason, I'll address this separately.
I'm closing this bug now.

[Olivier Tilloy (osomon)]

And the hint to force-badtest kopano-webapp has now been removed (https://bazaar.launchpad.net/~ubuntu-release/britney/hints-ubuntu/revision/3749), thanks Laney.