bootstrap fails with TLS handshake timeout

[Heather Lanigan]

https://jenkins.juju.canonical.com/job/nw-deploy-client-macos/2381/console

2020-10-14 01:13:28 INFO juju --show-log bootstrap google/europe-west1 testing-osx-client1 --config /var/folders/_1/j7_4g0n94m7_pllg_373gtc80000gp/T/tmpd33Ura.yaml --constraints --default-model testing-osx-client1 --agent-version 2.8.6 --bootstrap-series xenial
01:13:29 INFO juju.cmd supercommand.go:54 running juju [2.8.6 257 bfc8c5fd6cbb91cd94ab37ae8c0e46e8f6bb325f gc go1.14.9]
01:13:29 INFO cmd authkeys.go:114 Adding contents of "/Users/jenkins/cloud-city/juju-homes/testing-osx-client1/ssh/juju_id_rsa.pub" to authorized-keys
01:13:29 INFO cmd authkeys.go:114 Adding contents of "/Users/jenkins/.ssh/id_rsa.pub" to authorized-keys
ERROR Get "https://www.googleapis.com/compute/v1/projects/gothic-list-89514?alt=json&prettyPrint=false": oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": net/http: TLS handshake timeout
2020-10-14 01:13:39 ERROR Command '('juju', '--show-log', 'bootstrap', 'google/europe-west1', 'testing-osx-client1', '--config', '/var/folders/_1/j7_4g0n94m7_pllg_373gtc80000gp/T/tmpd33Ura.yaml', '--constraints', '', '--default-model', 'testing-osx-client1', '--agent-version', u'2.8.6', '--bootstrap-series', 'xenial')' returned non-zero exit status 1
Traceback (most recent call last):
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/utility.py", line 426, in logged_exception
    yield
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/deploy_stack.py", line 1012, in handle_bootstrap_exceptions
    yield
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/deploy_stack.py", line 968, in bootstrap_context
    yield
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/deploy_stack.py", line 1166, in new_bootstrap
    upload_tools, self.series, kwargs,
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/deploy_stack.py", line 595, in create_initial_model
    **boot_kwargs)
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/jujupy/client.py", line 1046, in bootstrap
    retvar, ct = self.juju('bootstrap', args, include_e=False)
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/jujupy/client.py", line 1336, in juju
    model, check, timeout, extra_env, suppress_err=suppress_err)
  File "/Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/jujupy/backend.py", line 226, in juju
    rval = call_func(args, stderr=stderr)
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '('juju', '--show-log', 'bootstrap', 'google/europe-west1', 'testing-osx-client1', '--config', '/var/folders/_1/j7_4g0n94m7_pllg_373gtc80000gp/T/tmpd33Ura.yaml', '--constraints', '', '--default-model', 'testing-osx-client1', '--agent-version', u'2.8.6', '--bootstrap-series', 'xenial')' returned non-zero exit status 1
2020-10-14 01:13:39 INFO /usr/local/opt/python@2/bin/python2.7 /Users/jenkins/workspace/nw-deploy-client-macos/acceptancetests/jujupy/timeout.py 1200.00 -- juju --show-log kill-controller testing-osx-client1 -y
01:13:40 INFO juju.cmd supercommand.go:54 running juju [2.8.6 257 bfc8c5fd6cbb91cd94ab37ae8c0e46e8f6bb325f gc go1.14.9]
ERROR controller testing-osx-client1 not found

[Pen Gale (pengale)]

Adding to 2.8.7 milestone. This is part of the work to address green in the CI.

[John A Meinel (jameinel)]

Is this something that we aren't retrying if there is a failure?

[John A Meinel (jameinel)]

Also, it seems like an oversight that:
01:13:29 INFO cmd authkeys.go:114 Adding contents of "/Users/jenkins/.ssh/id_rsa.pub" to authorized-keys
ERROR Get...

The ERROR line doesn't have a timestamp. So you can't tell if there is 10s between the INFO line and the ERROR, or 20min between the two.

[Simon Richardson (simonrichardson)]

Juju PR: https://github.com/juju/juju/pull/13024