charm download behind the enterprise proxy fails
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | juju-core |
High
|
Michael Foord | ||
| | 1.21 |
Low
|
Unassigned | ||
| | 1.22 |
High
|
Michael Foord | ||
| | openstack-installer |
Fix Released
|
Critical
|
Adam Stokes | |
Bug Description
root@trusty-
root@trusty-
root@trusty-
2014-12-16 22:00:02 INFO juju.cmd supercommand.go:37 running juju [1.20.14-
2014-12-16 22:00:02 WARNING juju.cmd.juju common.go:54 ignoring environments.yaml: using bootstrap config in file "/root/
2014-12-16 22:00:02 INFO juju.environs open.go:176 environment info already exists; using New not Prepare
2014-12-16 22:00:02 INFO juju.environs.sync sync.go:142 using sync tools source: https:/
using sync tools source: https:/
2014-12-16 22:00:02 INFO juju.environs.sync sync.go:65 listing available tools
listing available tools
2014-12-16 22:00:02 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:04 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:86 found 177 tools
found 177 tools
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:90 found 18 recent tools (version 1.20.14)
found 18 recent tools (version 1.20.14)
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:96 listing target tools storage
listing target tools storage
2014-12-16 22:01:20 ERROR juju.cmd supercommand.go:323 gomaasapi: got error back from server: 503 Service Unavailable (<HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
A communication error occurred: "Operation timed out"
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
)
root@trusty-
root@trusty-
2014-12-16 22:11:27 INFO juju.cmd supercommand.go:37 running juju [1.20.14-
2014-12-16 22:11:27 WARNING juju.cmd.juju common.go:54 ignoring environments.yaml: using bootstrap config in file "/root/
2014-12-16 22:11:27 DEBUG juju.environs open.go:85 ConfigForName found bootstrap config map[string]
2014-12-16 22:11:27 DEBUG juju.environs.
2014-12-16 22:11:27 INFO juju.environs open.go:176 environment info already exists; using New not Prepare
2014-12-16 22:11:27 DEBUG juju.provider.maas environprovider
2014-12-16 22:11:28 INFO juju.environs.sync sync.go:142 using sync tools source: https:/
using sync tools source: https:/
2014-12-16 22:11:28 INFO juju.environs.sync sync.go:65 listing available tools
listing available tools
2014-12-16 22:11:28 DEBUG juju.environs.tools tools.go:46 no architecture specified when finding tools, looking for any
2014-12-16 22:11:28 DEBUG juju.environs.tools tools.go:57 no series specified when finding tools, looking for any
2014-12-16 22:11:28 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:28 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:29 DEBUG juju.environs.
2014-12-16 22:11:29 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:29 DEBUG juju.environs.
2014-12-16 22:11:29 DEBUG juju.environs.
2014-12-16 22:11:29 DEBUG juju.environs.
2014-12-16 22:11:29 DEBUG juju.environs.
2014-12-16 22:11:29 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:31 DEBUG juju.environs.
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:86 found 177 tools
found 177 tools
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:90 found 18 recent tools (version 1.20.14)
found 18 recent tools (version 1.20.14)
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:96 listing target tools storage
listing target tools storage
2014-12-16 22:11:31 DEBUG juju.environs.tools storage.go:35 reading v1.* tools
2014-12-16 22:12:46 ERROR juju.cmd supercommand.go:323 gomaasapi: got error back from server: 503 Service Unavailable (<HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
A communication error occurred: "Operation timed out"
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
)
root@trusty-
| Narinder Gupta (narindergupta) wrote : | #1 |
| Christian Reis (kiko) wrote : | #2 |
BTW, Narinder, pyjuju is dead. :-) juju-core is the new black.
| no longer affects: | juju |
| Dimiter Naydenov (dimitern) wrote : | #3 |
After spending some time with Narinder on-site to debug the problem it appears the solution is to add the following to the environments.yaml file:
http-proxy: http://
https-proxy: http://
no-proxy: 172.16.17.140 # maas IP - needed to list/fetch tools from MAAS file storage
*AND*
export http_proxy=<the same url>
export https_proxy=<ditto>
export no_proxy=<maas ip>
After than running sync-tools was working.
Now, only one of these options *should* be sufficient (setting the env vars or envs.yaml). Why both are needed I'm still investigating.
| Dimiter Naydenov (dimitern) wrote : | #4 |
It appears with 1.21-beta3 the situation is worse, as even the workaround described in the previous comment does not work.
Narinder will provide more details later.
| Changed in juju-core: | |
| status: | New → Confirmed |
| importance: | Undecided → High |
| milestone: | none → 1.21-beta5 |
| Ian Booth (wallyworld) wrote : | #5 |
Just to provide some extra information to comment #3:
- the http-proxy settings in environments.yaml are used to configure each node created by juju in the cloud
- the exported env settings are used by the machine used to bootstrap the environment or run sync tools
So, if sync tools is being run, then it should only be necessary to set the env variables. However, running behind a firewall would necessitate any deployed nodes also point to the proxy, hence the yaml settings are needed also.
| Changed in juju-core: | |
| milestone: | 1.21-beta5 → 1.23 |
| Narinder Gupta (narindergupta) wrote : | #6 |
I hit the same bug when i tried the auto pilot within the same environment. Behind proxy it refused to launch the JUJU and error was can not upload the tool. I have raised an issue with cloud-installer as well.
| Narinder Gupta (narindergupta) wrote : | #7 |
here is the cloud-installer bug https:/
| Changed in juju-core: | |
| status: | Confirmed → Triaged |
| tags: | added: proxy sync-tools |
| Adam Stokes (adam-stokes) wrote : | #8 |
Further information on what I'm seeing:
When attempting to `juju deploy mysql` behind a corporate proxy I get the following error:
ubuntu@
2015-01-27 16:07:22 DEBUG juju.conn api.go:187 trying cached API connection settings
2015-01-27 16:07:22 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.197:17070 192.168.
2015-01-27 16:07:22 INFO juju.state.api apiclient.go:242 dialing "wss://
2015-01-27 16:07:22 INFO juju.state.api apiclient.go:176 connection established to "wss://
2015-01-27 16:09:30 ERROR juju.cmd supercommand.go:323 Cannot access the charm store. Are you connected to the internet? Error details: Get https:/
However, I am able to query that URL when i manually set the proxy on the CLI
ubuntu@
{"cs:trusty/
My environments.yaml file contains:
environments:
local:
type: local
container: kvm
lxc-clone: true
authorized-
default-series: trusty
admin-secret: pass
http-proxy: http://
https-proxy: http://
no-proxy: localhost
Shouldn't juju pull from its environments.yaml file for proxy information when querying things such as store.juju.
| Adam Stokes (adam-stokes) wrote : | #9 |
One other thing is we use the juju api to do deployments so setting the http/s proxy environment variables won't work for us
| Adam Stokes (adam-stokes) wrote : | #10 |
I also tried running:
ubuntu@
Which _did_ sync the tools, however,
ubuntu@
2015-01-27 16:37:53 DEBUG juju.conn api.go:187 trying cached API connection settings
2015-01-27 16:37:53 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.197:17070 192.168.
2015-01-27 16:37:53 INFO juju.state.api apiclient.go:242 dialing "wss://
2015-01-27 16:37:53 INFO juju.state.api apiclient.go:176 connection established to "wss://
^[[6~2015-01-27 16:40:00 ERROR juju.cmd supercommand.go:323 Cannot access the charm store. Are you connected to the internet? Error details: Get https:/
Still failed.
| Changed in cloud-installer: | |
| milestone: | none → v0.22 |
| assignee: | nobody → Adam Stokes (adam-stokes) |
| importance: | Undecided → Critical |
| status: | New → Confirmed |
| tags: | added: cloud-installer |
| tags: | added: deploy |
| summary: |
- juju sync-tools behind the enterprise proxy fails + charm download behind the enterprise proxy fails |
| Dimiter Naydenov (dimitern) wrote : | #11 |
We're investigating in depth why this happens and charmstore downloads ignore the proxy settings.
| Changed in juju-core: | |
| assignee: | nobody → Dimiter Naydenov (dimitern) |
| Changed in juju-core: | |
| assignee: | Dimiter Naydenov (dimitern) → nobody |
| assignee: | nobody → Michael Foord (mfoord) |
| Michael Foord (mfoord) wrote : | #12 |
It looks to me like the juju code that fetches charms, inside github.
| Michael Foord (mfoord) wrote : | #13 |
Ok, so it's slightly more tangled. We already have a proxyupdater worker that should be setting the proxy environment variables for the whole machine based on the settings for the juju environment (which should be propagated from environments.yaml). Setting the proxy environment should be enough for the http.DefaultClient used by the CharmStore to use the proxy. So something in the chain is not working correctly.
| Michael Foord (mfoord) wrote : | #14 |
I can reproduce charm download failure using a combination of squid, ufw and bootstrapping with the manual provider to a kvm image (that is restricted to ufw to only accessing the internet - with exceptions for ssh and the apiserver - via the squid proxy running on the host).
| Michael Foord (mfoord) wrote : | #15 |
If we connect to the jujud process, and inspect the environment, http_proxy and https_proxy are *not* set. However, /home/ubuntu/
| Michael Foord (mfoord) wrote : | #16 |
/home/ubuntu/
| Michael Foord (mfoord) wrote : | #17 |
There is code in the proxyupdateworker, that should be run every time jujud starts, to set the environment variables. It looks like this isn't working. Currently debugging.
| Michael Foord (mfoord) wrote : | #18 |
Now I'm seeing apt-get update failures with the error "Cannot initiate the connection to 3128:80". I did *not* see this problem last week. It looks like juju is now incorrectly parsing the http-proxy (and https-proxy) values from environments.yaml.
| Michael Foord (mfoord) wrote : | #19 |
Ok, so it looks like they were put into the apt config incorrectly. apt *requires* proxy information to be http://<proxy>:port/ whereas the normal http-proxy variables can be <proxy>:port. I'll raise a separate issue for that.
| Michael Foord (mfoord) wrote : | #20 |
Changing code in proxyupdaterworker to unconditionally set environment variables (instead of just on first run as current logic is) allows me to add a charm to the environment (this failed before).
| Michael Foord (mfoord) wrote : | #21 |
I do still see an error from the deployed unit (charmrevisionw
| Michael Foord (mfoord) wrote : | #22 |
Moving the proxyupdater worker to be started before other workers seems to remove the problem with the charmrevisionwo
| Michael Foord (mfoord) wrote : | #23 |
Note: https:/
Builds and tests pass. Needs new tests (and more manual testing).
| Michael Foord (mfoord) wrote : | #24 |
Problem with charmrevisionworker still present (intermittently - so timing dependent it seems), but the changes seem good anyway. Adding tests.
| Michael Foord (mfoord) wrote : | #25 |
The discussed fix is landing now. https:/
It's still worth looking into why the "first" logic didn't work as expected as it *looks* correct. Also I'm continuing to investigate the charmrevision worker issue.
| Adam Stokes (adam-stokes) wrote : | #26 |
Thanks, is it possible to get this backported to 1.21, 1.22 as well?
| Dimiter Naydenov (dimitern) wrote : | #27 |
Yes Adam, I'll backport it to 1.21 and 1.22.
| Changed in juju-core: | |
| status: | Triaged → Fix Committed |
| Dimiter Naydenov (dimitern) wrote : | #28 |
Correction, so 1.21 does not use the same proxyupdater worker the same way, so it's not trivial to backport and we need to prioritize it if it's that important, but since we've already released 1.22 RC and 1.23 will be soon out I think backporting this should be a low priority.
| no longer affects: | juju-core/1.21 |
| Michael Foord (mfoord) wrote : | #29 |
The charmrevisionworker is started *after* the proxyupdater - so the fact that I *sometimes* see an error in the logs from the charmrevisionworker (unable to access the internet) is a simple race condition. Sometimes this worker is started before the proxy settings have been put in the environment. The right fix is for starting the proxyupdater to block until SetUp is completed.
The full call stack (traced in cmd/jujud/
MachineAgent
proxyupdater.New called from ->
postUpgradeAPIW
APIWorker
Run
charmrevisionworker created in ->
startEnvWorkers
StateWorker
stateStarter
newStateStarter
Run
And APIWorker is called before newStateStarter
| Dimiter Naydenov (dimitern) wrote : | #30 |
We won't fix this for 1.21 as the code changes between 1.21 and 1.22/1.23 are significant enough to make the effort moot in the face of 1.21.2 getting released (hopefully tomorrow) and the 1.22 release to follow shortly after (couple of weeks), which already has the fix. The only related fix we'll do for 1.21 is to improve the charmrevisionup
| Michael Foord (mfoord) wrote : | #31 |
The fix was backported to 1.22 incorrectly. Fixing now.
| Adam Stokes (adam-stokes) wrote : | #32 |
I tried 1.22-beta3 and was unable to provision any machines via local provider:
juju log: http://
environments.yaml: http://
I'll leave the machine up if you need anything else.
| Adam Stokes (adam-stokes) wrote : | #33 |
So just thinking out loud here, is it possible that during lxc create it is still having trouble getting the lxc image since it does a call out to wget? Are we passing any proxy information through to lxc during creation? What about for KVM? I am able to bootstrap our single installer which uses KVM intially, but then the containers within that environment fail to create/startup, just wondering if we're missing proxy settings there as well.
| Michael Foord (mfoord) wrote : | #34 |
I didn't look at container creation. It's highly likely that we don't pass proxy settings through to the container. wget *should* use proxy settings from the environment on the host machine though. It would be better to file "local provider fails behind a proxy" as a new bug though as this one was about charm download and is at "fix committed/released" stage.
| Changed in juju-core: | |
| status: | Fix Committed → Fix Released |
| Adam Stokes (adam-stokes) wrote : | #35 |
FYI, I did get this to work finally, I didn't have all the required ips in the no-proxy setting.
| Changed in cloud-installer: | |
| status: | Confirmed → Fix Committed |
| Changed in cloud-installer: | |
| status: | Fix Committed → Fix Released |
| Changed in juju-core: | |
| milestone: | 1.23 → 1.23-beta1 |
due to this bug I can not prepare any demo at HP for JUJU. MAAS deployment behind proxy works great and one point of time this used to work but now it is not working.