1.25.0: bootstrap failure - WARNING discarding API open error: EOF
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
Cheryl Jennings | ||
juju-core |
Fix Released
|
High
|
Cheryl Jennings | ||
1.25 |
Fix Released
|
High
|
Cheryl Jennings |
Bug Description
We have hit this error multiple times with Juju 1.25.0:
Bootstrapping environment "ci-oil-slave13"
Starting new instance for initial state server
Launching instance
- /MAAS/api/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to hayward-24.oil:22
Attempting to connect to hayward-24.oil:22
Attempting to connect to 10.245.0.153:22
Warning: Permanently added '10.245.0.153' (ECDSA) to the list of known hosts.
Logging to /var/log/
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https:/
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap agent installed
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
Waiting for API to become available
WARNING discarding API open error: EOF
ERROR EOF
2016-01-26 15:07:36,633 [ERROR] oil_ci.juju.client: Calling "juju bootstrap" failed!
2016-01-26 15:07:36,635 [ERROR] oil_ci.cli.api: Deployment failed
Traceback (most recent call last):
File "/usr/lib/
rc = deploy.deploy()
File "/usr/lib/
self.
File "/usr/lib/
constraints
File "/usr/lib/
raise JujuTestClientE
JujuTestClientE
+ rc=1
+ echo 'Deployment returned: 1'
Deployment returned: 1
Please note that this one looks like bug 1338511 but that one was fix released, so opening a new bug. Unless it's a regression, this would be a new issue.
Changed in juju-core: | |
milestone: | 1.25.4 → 2.0-alpha3 |
Changed in juju-core: | |
milestone: | 2.0-alpha3 → 2.0-beta3 |
Changed in juju-core: | |
assignee: | nobody → Cheryl Jennings (cherylj) |
Changed in juju-core: | |
milestone: | 2.0-beta3 → 2.0-beta4 |
Changed in juju-core: | |
status: | In Progress → Fix Committed |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
affects: | juju-core → juju |
Changed in juju: | |
milestone: | 2.0-beta4 → none |
milestone: | none → 2.0-beta4 |
Changed in juju-core: | |
assignee: | nobody → Cheryl Jennings (cherylj) |
importance: | Undecided → High |
status: | New → Fix Released |
I think what's happening here is that the check in bootstrap where we wait for the API to come up isn't complete. The check will try to make an arbitrary call into the API 60 times, with a one second delay. If the error message received from that call is either that an upgrade is in progress, connection shut down, or EOF, it keeps retrying to make the call.
However, it looks like in the above output, the EOF error is not coming from the API call, but rather from the attempt to open / login to the API, which doesn't have the same check to see if the error returned is one of the three errors above.
I think what we need to do is keep retrying if the call to open / login to the API connection also returns EOF or connection shutdown.