juju-core

charm download behind the enterprise proxy fails

Bug #1403225 reported by Narinder Gupta
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Michael Foord
juju-core 1.23-beta1
1.21
Won't Fix
Low
Unassigned
1.22
Fix Released
High
Michael Foord
juju-core 1.22-beta3
openstack-installer
Fix Released
Critical
Adam Stokes

Bug Description

root@trusty-mas-vm:~#
root@trusty-mas-vm:~#
root@trusty-mas-vm:~# http_proxy=http://<proxy>:8080 https_proxy=http://<proxy>:8080 juju sync-tools --show-log
2014-12-16 22:00:02 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2014-12-16 22:00:02 WARNING juju.cmd.juju common.go:54 ignoring environments.yaml: using bootstrap config in file "/root/.juju/environments/maas.jenv"
2014-12-16 22:00:02 INFO juju.environs open.go:176 environment info already exists; using New not Prepare
2014-12-16 22:00:02 INFO juju.environs.sync sync.go:142 using sync tools source: https://streams.canonical.com/juju/tools
using sync tools source: https://streams.canonical.com/juju/tools
2014-12-16 22:00:02 INFO juju.environs.sync sync.go:65 listing available tools
listing available tools
2014-12-16 22:00:02 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:04 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:86 found 177 tools
found 177 tools
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:90 found 18 recent tools (version 1.20.14)
found 18 recent tools (version 1.20.14)
2014-12-16 22:00:05 INFO juju.environs.sync sync.go:96 listing target tools storage
listing target tools storage
2014-12-16 22:01:20 ERROR juju.cmd supercommand.go:323 gomaasapi: got error back from server: 503 Service Unavailable (<HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
A communication error occurred: "Operation timed out"
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
)
root@trusty-mas-vm:~#
root@trusty-mas-vm:~# http_proxy=http://proxy.houston.hp.com:8080 https_proxy=http://proxy.houston.hp.com:8080 juju sync-tools --show-log --debug
2014-12-16 22:11:27 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2014-12-16 22:11:27 WARNING juju.cmd.juju common.go:54 ignoring environments.yaml: using bootstrap config in file "/root/.juju/environments/maas.jenv"
2014-12-16 22:11:27 DEBUG juju.environs open.go:85 ConfigForName found bootstrap config map[string]interface {}{"logging-config":"<root>=INFO;unit=DEBUG", "maas-server":"http://172.16.17.140/MAAS/", "bootstrap-retry-delay":5, "syslog-port":6514, "tools-url":"", "api-port":17070, "authorized-keys":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLrDG9kX0pxp2XLuDWf0Ohe7M3o0RMJ6GJuKvmhreLKGCJnEU+RrDPoCNUTNRgyP95hdezCizFtFWL7QPGB78Sauqp7o5zrbNrZQwL3tknbXneRZGriwKVbJtkqD90nw4DrUl68de4iwh/M/PPeDcQpMqFdhlg80hMlSMseoKeouXO3mr2sRc8YsroeY+5CGeKOCY5cgoshFw+NFSE4byhTwG8rHXdlef3Gh6Ba14xfXncUO4BZuIYBGYTMy95WtwS2Su7R85b7vDyInQxQSj0hFh57BhhpG3s1IfU58ePD6BpvR81CMn/M181XXqzONeFtGBsg/anCoKNHs8sRwSD juju-client-key\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYomHwbNjXW5raBK0dQdSNgWBm3JpO3cqhOYoj7+Q3UQAHc/DeY5HG3nJQ/AAqVoqwdMahVkGxKOZHye1wLzvTfQUxWNVxejtnlqlq5GMXvQYSfIo+kssqqQjjbK+m1Ub6hXqAIKDBI6BpEutRZJVvBHUkHrXpyoxMpvP5rxgEYrRm7AfMDeAVa6oZ3XqNv5+aVF/nC0FcQi88Nhiu34urqjEN91LW+lBAE4avOlmyxkRT4L+YaQQwNpmaM0Uiv+FNch4bw3LYTgycJlD6cqrTSIg7Av4Orz4JVwZrtUeI+SmtE9euIWO7weAyC64TM6XG3lkWAMY41C2Dp73/ECZD root@trusty-mas-vm\n", "firewall-mode":"instance", "maas-oauth":"yy58kgtN7nhUfUnvVf:xsC7nKdCNJgnrVSdUb:dzk6VZP5zUQgu6mhffJ8an6nujKKGGk7", "type":"maas", "ca-cert":"-----BEGIN CERTIFICATE-----\nMIICWTCCAcSgAwIBAgIBADALBgkqhkiG9w0BAQUwQjENMAsGA1UEChMEanVqdTEx\nMC8GA1UEAwwoanVqdS1nZW5lcmF0ZWQgQ0EgZm9yIGVudmlyb25tZW50ICJtYWFz\nIjAeFw0xNDEyMTYyMTU0MThaFw0yNDEyMTYyMTU5MThaMEIxDTALBgNVBAoTBGp1\nanUxMTAvBgNVBAMMKGp1anUtZ2VuZXJhdGVkIENBIGZvciBlbnZpcm9ubWVudCAi\nbWFhcyIwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANlYuJq8or5OukqcOZ5A\nkiwrOPQrYOzR+KrxQB52kvQ+VT7mcqjhN+WawIha422ZulYxppLhurdcmuBAcTpG\nC7Bqj6pS+6g2SkYAwcjHnYPiI4/nnAk+CP5rswQl/5bItsVOkbvQ3olI/5nTtpwa\nekYRxUaSZa5OES7h5mPrMaA/AgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIApDAPBgNV\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRfGaHVe8RNTJ/Fz3UkhGJMteGRQTAfBgNV\nHSMEGDAWgBRfGaHVe8RNTJ/Fz3UkhGJMteGRQTALBgkqhkiG9w0BAQUDgYEACngb\nVx/lFhU7EfT7qN2VC5XIURbJJljI8DQrz59hDMuS4ezC8p5JYy/+ApfER05qOuDq\nRD/Pr6lhsFwM1veDo5ghpLzOZOKnS0CMfIpa+LRvWqQQTNPcFAJnQHbwymRmhfHK\nEWmADWYgdP60Rq4U/wzFqN4PLmEn2eUsSjxzD8E=\n-----END CERTIFICATE-----\n", "disable-network-management":false, "image-metadata-url":"", "admin-secret":"admin", "maas-agent-name":"6e66815d-e5be-4074-8251-197fc9a1f6fc", "name":"maas", "proxy-ssh":true, "state-port":37017, "tools-metadata-url":"", "bootstrap-timeout":1800, "ca-private-key":"-----BEGIN RSA PRIVATE KEY-----\nMIICXgIBAAKBgQDZWLiavKK+TrpKnDmeQJIsKzj0K2Ds0fiq8UAedpL0PlU+5nKo\n4TflmsCIWuNtmbpWMaaS4bq3XJrgQHE6Rguwao+qUvuoNkpGAMHIx52D4iOP55wJ\nPgj+a7MEJf+WyLbFTpG70N6JSP+Z07acGnpGEcVGkmWuThEu4eZj6zGgPwIDAQAB\nAoGBAI11+FkGE6pnlUqAWeQNRNYon54UKbaZhw7DjKcHJ+h+NX+F//S4+WegjFVz\nSnkISEZXdII/FwbG6Cqy5r813d6WmSxBUVWNN8g4+F4vbcWKdAgeKvbANxoJIpta\nZYkfwnWNFe98Fr1gILF5jpU0EhcBpUwud4HpF82bpM5bwhUJAkEA80dYhwHrzJEy\nHQv/vUyM+BPZwBKWnrR/SLw17XQYYf5vQr1NGcR2UIztWfP94nv5FgugCuZJbESc\nfgKIGXPRbQJBAOS2PAUW8NWBIj2r0M7BmPUPFwaWYDI+wE0Mu4y8kOiWCO2H1guc\nD5STAD66TRvO0j7NwEQHb9+ZJUQDS5WgWNsCQQCIHz2bnSmRNxTxR8t64iBmlsTa\nQgxhFOG7kwQhrzoNas5aa41+g30oKRKNqIZAedmvyiVss9oVaHaf007P3wwpAkAo\n4THFLx6I6vX10/oUmqH+W14gPUygbnrsyrPTKfk0uWyLGFZ36MMYKVZH4ADcUZFe\nzGPrRPEIJFV23Kfr7Kc1AkEAn5QoLacZsFxBB80Di4E16y0qokN2g02NgTgiicKR\niNq0YuVKBf6e3C7QF3ZC3xoZor8jps2HBm1ra+LYEf5ufA==\n-----END RSA PRIVATE KEY-----\n", "charm-store-auth":"", "image-stream":"", "ssl-hostname-verification":true, "test-mode":false, "bootstrap-addresses-delay":10, "default-series":"trusty", "development":false, "lxc-clone-aufs":false}
2014-12-16 22:11:27 DEBUG juju.environs.configstore disk.go:69 Making /root/.juju/environments
2014-12-16 22:11:27 INFO juju.environs open.go:176 environment info already exists; using New not Prepare
2014-12-16 22:11:27 DEBUG juju.provider.maas environprovider.go:30 opening environment "maas".
2014-12-16 22:11:28 INFO juju.environs.sync sync.go:142 using sync tools source: https://streams.canonical.com/juju/tools
using sync tools source: https://streams.canonical.com/juju/tools
2014-12-16 22:11:28 INFO juju.environs.sync sync.go:65 listing available tools
listing available tools
2014-12-16 22:11:28 DEBUG juju.environs.tools tools.go:46 no architecture specified when finding tools, looking for any
2014-12-16 22:11:28 DEBUG juju.environs.tools tools.go:57 no series specified when finding tools, looking for any
2014-12-16 22:11:28 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:28 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:29 DEBUG juju.environs.simplestreams simplestreams.go:606 using default candidate for content id "com.ubuntu.juju:released:tools" are {20141216 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
2014-12-16 22:11:29 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:29 DEBUG juju.environs.simplestreams simplestreams.go:477 no mirror information available for { }: mirror info with cloud { } not found
2014-12-16 22:11:29 DEBUG juju.environs.simplestreams simplestreams.go:397 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index.sjson"
2014-12-16 22:11:29 DEBUG juju.environs.simplestreams simplestreams.go:564 candidate matches for products ["com.ubuntu.juju:12.04:amd64" "com.ubuntu.juju:12.04:i386" "com.ubuntu.juju:12.04:armhf" "com.ubuntu.juju:12.04:arm64" "com.ubuntu.juju:12.04:ppc64el" "com.ubuntu.juju:12.10:amd64" "com.ubuntu.juju:12.10:i386" "com.ubuntu.juju:12.10:armhf" "com.ubuntu.juju:12.10:arm64" "com.ubuntu.juju:12.10:ppc64el" "com.ubuntu.juju:13.04:amd64" "com.ubuntu.juju:13.04:i386" "com.ubuntu.juju:13.04:armhf" "com.ubuntu.juju:13.04:arm64" "com.ubuntu.juju:13.04:ppc64el" "com.ubuntu.juju:13.10:amd64" "com.ubuntu.juju:13.10:i386" "com.ubuntu.juju:13.10:armhf" "com.ubuntu.juju:13.10:arm64" "com.ubuntu.juju:13.10:ppc64el" "com.ubuntu.juju:14.04:amd64" "com.ubuntu.juju:14.04:i386" "com.ubuntu.juju:14.04:armhf" "com.ubuntu.juju:14.04:arm64" "com.ubuntu.juju:14.04:ppc64el" "com.ubuntu.juju:14.10:amd64" "com.ubuntu.juju:14.10:i386" "com.ubuntu.juju:14.10:armhf" "com.ubuntu.juju:14.10:arm64" "com.ubuntu.juju:14.10:ppc64el" "com.ubuntu.juju:15.04:amd64" "com.ubuntu.juju:15.04:i386" "com.ubuntu.juju:15.04:armhf" "com.ubuntu.juju:15.04:arm64" "com.ubuntu.juju:15.04:ppc64el"] are [{Tue, 16 Dec 2014 22:02:06 +0000 products:1.0 content-download [] streams/v1/com.ubuntu.juju:released:tools.sjson [com.ubuntu.juju:12.04:amd64 com.ubuntu.juju:12.04:armhf com.ubuntu.juju:12.04:i386 com.ubuntu.juju:12.10:amd64 com.ubuntu.juju:12.10:i386 com.ubuntu.juju:13.04:amd64 com.ubuntu.juju:13.04:i386 com.ubuntu.juju:13.10:amd64 com.ubuntu.juju:13.10:armhf com.ubuntu.juju:13.10:i386 com.ubuntu.juju:14.04:amd64 com.ubuntu.juju:14.04:arm64 com.ubuntu.juju:14.04:armhf com.ubuntu.juju:14.04:i386 com.ubuntu.juju:14.04:powerpc com.ubuntu.juju:14.04:ppc64 com.ubuntu.juju:14.04:ppc64el com.ubuntu.juju:14.10:amd64 com.ubuntu.juju:14.10:arm64 com.ubuntu.juju:14.10:armhf com.ubuntu.juju:14.10:i386 com.ubuntu.juju:14.10:ppc64 com.ubuntu.juju:14.10:ppc64el com.ubuntu.juju:15.04:amd64 com.ubuntu.juju:15.04:arm64 com.ubuntu.juju:15.04:armhf com.ubuntu.juju:15.04:i386 com.ubuntu.juju:15.04:ppc64 com.ubuntu.juju:15.04:ppc64el]}]
2014-12-16 22:11:29 DEBUG juju.environs.simplestreams simplestreams.go:877 finding products at path "streams/v1/com.ubuntu.juju:released:tools.sjson"
2014-12-16 22:11:29 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-12-16 22:11:31 DEBUG juju.environs.simplestreams simplestreams.go:915 metadata: &{map[com.ubuntu.juju:14.10:arm64:{ 1.18.4 arm64 map[20141216:0xc2101e0a80]} com.ubuntu.juju:15.04:arm64:{ 1.20.12 arm64 map[20141216:0xc210113900]} com.ubuntu.juju:12.04:amd64:{ 1.15.0 amd64 map[20141216:0xc2101ec2a0]} com.ubuntu.juju:13.10:amd64:{ 1.15.0 amd64 map[20141216:0xc2101ecc60]} com.ubuntu.juju:14.04:ppc64el:{ 1.17.2 ppc64el map[20141216:0xc2101e0420]} com.ubuntu.juju:12.10:amd64:{ 1.15.0 amd64 map[20141216:0xc2101ec600]} com.ubuntu.juju:12.10:i386:{ 1.15.0 i386 map[20141216:0xc2101ec720]} com.ubuntu.juju:13.04:amd64:{ 1.15.0 amd64 map[20141216:0xc2101ec840]} com.ubuntu.juju:13.04:i386:{ 1.15.0 i386 map[20141216:0xc2101ecb40]} com.ubuntu.juju:14.04:ppc64:{ 1.18.4 ppc64 map[20141216:0xc2101bdd80]} com.ubuntu.juju:12.04:i386:{ 1.15.0 i386 map[20141216:0xc2101ec4e0]} com.ubuntu.juju:13.10:i386:{ 1.15.0 i386 map[20141216:0xc2101ecea0]} com.ubuntu.juju:14.04:amd64:{ 1.16.2 amd64 map[20141216:0xc2101bd000]} com.ubuntu.juju:14.04:arm64:{ 1.17.2 arm64 map[20141216:0xc2101bd5a0]} com.ubuntu.juju:14.04:armhf:{ 1.16.2 armhf map[20141216:0xc2101bd720]} com.ubuntu.juju:14.04:powerpc:{ 1.17.2 powerpc map[20141216:0xc2101bdba0]} com.ubuntu.juju:14.10:amd64:{ 1.18.2 amd64 map[20141216:0xc2101e06c0]} com.ubuntu.juju:14.10:armhf:{ 1.18.2 armhf map[20141216:0xc2101e0ea0]} com.ubuntu.juju:14.10:i386:{ 1.18.2 i386 map[20141216:0xc210113060]} com.ubuntu.juju:15.04:amd64:{ 1.20.12 amd64 map[20141216:0xc2101135a0]} com.ubuntu.juju:15.04:armhf:{ 1.20.12 armhf map[20141216:0xc210113ae0]} com.ubuntu.juju:15.04:i386:{ 1.20.12 i386 map[20141216:0xc21011a0c0]} com.ubuntu.juju:12.04:armhf:{ 1.16.2 armhf map[20141216:0xc2101ec3c0]} com.ubuntu.juju:13.10:armhf:{ 1.16.2 armhf map[20141216:0xc2101ecd80]} com.ubuntu.juju:14.04:i386:{ 1.16.2 i386 map[20141216:0xc2101bd9c0]} com.ubuntu.juju:14.10:ppc64:{ 1.18.4 ppc64 map[20141216:0xc210113180]} com.ubuntu.juju:15.04:ppc64:{ 1.20.12 ppc64 map[20141216:0xc21011a240]} com.ubuntu.juju:14.10:ppc64el:{ 1.18.4 ppc64el map[20141216:0xc210113480]} com.ubuntu.juju:15.04:ppc64el:{ 1.20.12 ppc64el map[20141216:0xc21011a360]}] map[] Tue, 16 Dec 2014 22:02:06 +0000 products:1.0 com.ubuntu.juju:released:tools }
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:86 found 177 tools
found 177 tools
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:90 found 18 recent tools (version 1.20.14)
found 18 recent tools (version 1.20.14)
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-precise-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-precise-amd64.tgz 932640702b5d9f08a312118a4afe330444308f92b7de350de9e547719f084bd9 8130412}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-precise-armhf https://streams.canonical.com/juju/tools/releases/juju-1.20.14-precise-armhf.tgz bb41ba9604bbee04084577bc1db61d3011c51ad9fca99659fe7548a9f14498c1 7769053}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-precise-i386 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-precise-i386.tgz b33a5a24614d7424ced74215b8951e80018d173c508ee08adc3d89b884e35a4c 7861992}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-trusty-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-trusty-amd64.tgz ad7360aa5512b8d7cc1ad0ceb223bd9a1c8f5c6b0b131b5a3bd8fde623b85585 8130261}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-trusty-arm64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-trusty-arm64.tgz 2422b8d61ec6e5792d6cb7b46866c91db56c620271ae74bce27d99d08e5ccf91 8930982}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-trusty-armhf https://streams.canonical.com/juju/tools/releases/juju-1.20.14-trusty-armhf.tgz 2fdf14f399db9b7d1bc8fd91ee19dae4b0143ed725d6f75fec8df0297268e9ff 7768909}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-trusty-i386 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-trusty-i386.tgz 4f3ab4574c66eee2e0d7693e0e913f23b12621f1292a1f1c896be1339de3c9f5 7862574}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-trusty-ppc64el https://streams.canonical.com/juju/tools/releases/juju-1.20.14-trusty-ppc64el.tgz 82ad61c4ce660bed76ea80a28028dcc43bb3cf30e610fafdd3562df6d12d6e03 8774066}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-utopic-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-utopic-amd64.tgz e4ad32e7050924a28c788de593dbf28a3ec8d9d8122524a37ee49beabf762164 8130261}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-utopic-arm64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-utopic-arm64.tgz 033733937fc7cf8e0ec70b4656c233a771d7cf4571a84140dbeffb0ae78e6e4e 8869675}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-utopic-armhf https://streams.canonical.com/juju/tools/releases/juju-1.20.14-utopic-armhf.tgz 956b91b2994a78bd319b80b80bf6459909f47772dbbfaf88c98f215cdd7fa513 7768909}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-utopic-i386 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-utopic-i386.tgz d3fd1121754ee10cde3062d0b179ccf61b4d93973b7e2b6889a0b972892660b6 7862574}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-utopic-ppc64el https://streams.canonical.com/juju/tools/releases/juju-1.20.14-utopic-ppc64el.tgz 29f5662800bd4d04b6b0535481fd2f011c65c56cf476996dce7f984a15a16b49 8779288}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-vivid-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-vivid-amd64.tgz 53b4b313b8a96b64c3a41011f2faa911c146489d43445e15fb4ff54fce3b7a75 7144743}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-vivid-arm64 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-vivid-arm64.tgz 3da052cb71d7ca38653bf98b72f2a67f663cd5701efb8f77867b537a67854a96 8862936}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-vivid-armhf https://streams.canonical.com/juju/tools/releases/juju-1.20.14-vivid-armhf.tgz 9336211248526173d334faf203c7f5501e091f62e7002f30deca3c85cc4e6c40 6576898}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-vivid-i386 https://streams.canonical.com/juju/tools/releases/juju-1.20.14-vivid-i386.tgz 2287b92a2d0d6188910530a455cd61b42f63d8514119726070eb81c5a61245cd 6882012}
2014-12-16 22:11:31 DEBUG juju.environs.sync sync.go:93 found source tool: &{1.20.14-vivid-ppc64el https://streams.canonical.com/juju/tools/releases/juju-1.20.14-vivid-ppc64el.tgz 75d212c40b6ad4c7fe14bbf10577356ae66e3c7ae7579d9b1f4138585981b110 8779298}
2014-12-16 22:11:31 INFO juju.environs.sync sync.go:96 listing target tools storage
listing target tools storage
2014-12-16 22:11:31 DEBUG juju.environs.tools storage.go:35 reading v1.* tools
2014-12-16 22:12:46 ERROR juju.cmd supercommand.go:323 gomaasapi: got error back from server: 503 Service Unavailable (<HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
A communication error occurred: "Operation timed out"
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
)
root@trusty-mas-vm:~#

Revision history for this message
Narinder Gupta (narindergupta) wrote :

due to this bug I can not prepare any demo at HP for JUJU. MAAS deployment behind proxy works great and one point of time this used to work but now it is not working.

Revision history for this message
Christian Reis (kiko) wrote :

BTW, Narinder, pyjuju is dead. :-) juju-core is the new black.

Michael Foord (mfoord)
no longer affects: juju
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

After spending some time with Narinder on-site to debug the problem it appears the solution is to add the following to the environments.yaml file:

http-proxy: http://proxy.url.address:8080/ # needed to list/fetch tools from http://streams.canonical.com
https-proxy: http://proxy.url.address:8080/ # so it will work for https://streams.canonical.com/ as well
no-proxy: 172.16.17.140 # maas IP - needed to list/fetch tools from MAAS file storage

*AND*

export http_proxy=<the same url>
export https_proxy=<ditto>
export no_proxy=<maas ip>

After than running sync-tools was working.
Now, only one of these options *should* be sufficient (setting the env vars or envs.yaml). Why both are needed I'm still investigating.

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

It appears with 1.21-beta3 the situation is worse, as even the workaround described in the previous comment does not work.
Narinder will provide more details later.

Changed in juju-core:
status: New → Confirmed
importance: Undecided → High
milestone: none → 1.21-beta5
Revision history for this message
Ian Booth (wallyworld) wrote :

Just to provide some extra information to comment #3:

- the http-proxy settings in environments.yaml are used to configure each node created by juju in the cloud
- the exported env settings are used by the machine used to bootstrap the environment or run sync tools

So, if sync tools is being run, then it should only be necessary to set the env variables. However, running behind a firewall would necessitate any deployed nodes also point to the proxy, hence the yaml settings are needed also.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.21-beta5 → 1.23
Revision history for this message
Narinder Gupta (narindergupta) wrote :

I hit the same bug when i tried the auto pilot within the same environment. Behind proxy it refused to launch the JUJU and error was can not upload the tool. I have raised an issue with cloud-installer as well.

Revision history for this message
Narinder Gupta (narindergupta) wrote :

here is the cloud-installer bug https://github.com/Ubuntu-Solutions-Engineering/openstack-installer/issues/335 for references.

Curtis Hovey (sinzui)
Changed in juju-core:
status: Confirmed → Triaged
Curtis Hovey (sinzui)
tags: added: proxy sync-tools
Revision history for this message
Adam Stokes (adam-stokes) wrote :

Further information on what I'm seeing:

When attempting to `juju deploy mysql` behind a corporate proxy I get the following error:

ubuntu@uoi-bootstrap:~$ JUJU_HOME=~/.cloud-install/juju/ juju deploy mysql --debug2015-01-27 16:07:22 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2015-01-27 16:07:22 DEBUG juju.conn api.go:187 trying cached API connection settings
2015-01-27 16:07:22 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.197:17070 192.168.122.1:17070]
2015-01-27 16:07:22 INFO juju.state.api apiclient.go:242 dialing "wss://localhost:17070/environment/d507d300-dc0f-47e8-8b8f-ce8a8548fafc/api"
2015-01-27 16:07:22 INFO juju.state.api apiclient.go:176 connection established to "wss://localhost:17070/environment/d507d300-dc0f-47e8-8b8f-ce8a8548fafc/api"
2015-01-27 16:09:30 ERROR juju.cmd supercommand.go:323 Cannot access the charm store. Are you connected to the internet? Error details: Get https://store.juju.ubuntu.com/charm-info?charms=cs%3Atrusty%2Fmysql: dial tcp 91.189.95.66:443: connection timed out

However, I am able to query that URL when i manually set the proxy on the CLI

ubuntu@uoi-bootstrap:~$ http_proxy=http://squid.internal:3128 https_proxy=http://squid.internal:3128 curl -L https://store.juju.ubuntu.com/charm-info?charms=cs%3Atrusty%2Fmysql
{"cs:trusty/mysql":{"canonical-url":"cs:trusty/mysql","revision":14,"sha256":"2a66021a581024312968fd6e5803da7faf7e9da7def58903dd6a9f68d1e68c0f","digest":"<email address hidden>"}}

My environments.yaml file contains:
environments:
  local:
    type: local
    container: kvm
    lxc-clone: true
    authorized-keys-path: ~/.ssh/id_rsa.pub
    default-series: trusty
    admin-secret: pass
    http-proxy: http://squid.internal:3128
    https-proxy: http://squid.internal:3128
    no-proxy: localhost

Shouldn't juju pull from its environments.yaml file for proxy information when querying things such as store.juju.ubuntu.com?

Revision history for this message
Adam Stokes (adam-stokes) wrote :

One other thing is we use the juju api to do deployments so setting the http/s proxy environment variables won't work for us

Revision history for this message
Adam Stokes (adam-stokes) wrote :

I also tried running:

ubuntu@uoi-bootstrap:~$ http_proxy=http://squid.internal:3128 https_proxy=http://squid.internal:3128 JUJU_HOME=~/.cloud-install/juju juju sync-tools

Which _did_ sync the tools, however,

ubuntu@uoi-bootstrap:~$ JUJU_HOME=~/.cloud-install/juju/ juju deploy mysql --debug2015-01-27 16:37:53 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2015-01-27 16:37:53 DEBUG juju.conn api.go:187 trying cached API connection settings
2015-01-27 16:37:53 INFO juju.conn api.go:270 connecting to API addresses: [localhost:17070 10.0.3.197:17070 192.168.122.1:17070]
2015-01-27 16:37:53 INFO juju.state.api apiclient.go:242 dialing "wss://localhost:17070/environment/d507d300-dc0f-47e8-8b8f-ce8a8548fafc/api"
2015-01-27 16:37:53 INFO juju.state.api apiclient.go:176 connection established to "wss://localhost:17070/environment/d507d300-dc0f-47e8-8b8f-ce8a8548fafc/api"
^[[6~2015-01-27 16:40:00 ERROR juju.cmd supercommand.go:323 Cannot access the charm store. Are you connected to the internet? Error details: Get https://store.juju.ubuntu.com/charm-info?charms=cs%3Atrusty%2Fmysql: dial tcp 91.189.95.67:443: connection timed out

Still failed.

Changed in cloud-installer:
milestone: none → v0.22
assignee: nobody → Adam Stokes (adam-stokes)
importance: Undecided → Critical
status: New → Confirmed
Adam Stokes (adam-stokes)
tags: added: cloud-installer
Curtis Hovey (sinzui)
tags: added: deploy
Ian Booth (wallyworld)
summary: - juju sync-tools behind the enterprise proxy fails
+ charm download behind the enterprise proxy fails
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

We're investigating in depth why this happens and charmstore downloads ignore the proxy settings.

Changed in juju-core:
assignee: nobody → Dimiter Naydenov (dimitern)
Dimiter Naydenov (dimitern)
Changed in juju-core:
assignee: Dimiter Naydenov (dimitern) → nobody
assignee: nobody → Michael Foord (mfoord)
Revision history for this message
Michael Foord (mfoord) wrote :

It looks to me like the juju code that fetches charms, inside github.com/juju/charms , is doing straight http(s) requests without using proxy information. Updating charms.CharmStore to allow the setting of proxy information should fix this problem.

Revision history for this message
Michael Foord (mfoord) wrote :

Ok, so it's slightly more tangled. We already have a proxyupdater worker that should be setting the proxy environment variables for the whole machine based on the settings for the juju environment (which should be propagated from environments.yaml). Setting the proxy environment should be enough for the http.DefaultClient used by the CharmStore to use the proxy. So something in the chain is not working correctly.

Revision history for this message
Michael Foord (mfoord) wrote :

I can reproduce charm download failure using a combination of squid, ufw and bootstrapping with the manual provider to a kvm image (that is restricted to ufw to only accessing the internet - with exceptions for ssh and the apiserver - via the squid proxy running on the host).

Revision history for this message
Michael Foord (mfoord) wrote :

If we connect to the jujud process, and inspect the environment, http_proxy and https_proxy are *not* set. However, /home/ubuntu/.juju-proxy *has* been written out correctly. So the environments file has been written, it just isn't set in the environment.

Revision history for this message
Michael Foord (mfoord) wrote :

/home/ubuntu/.profile has been set to source the .juju-proxy file.

Revision history for this message
Michael Foord (mfoord) wrote :

There is code in the proxyupdateworker, that should be run every time jujud starts, to set the environment variables. It looks like this isn't working. Currently debugging.

Revision history for this message
Michael Foord (mfoord) wrote :

Now I'm seeing apt-get update failures with the error "Cannot initiate the connection to 3128:80". I did *not* see this problem last week. It looks like juju is now incorrectly parsing the http-proxy (and https-proxy) values from environments.yaml.

Revision history for this message
Michael Foord (mfoord) wrote :

Ok, so it looks like they were put into the apt config incorrectly. apt *requires* proxy information to be http://<proxy>:port/ whereas the normal http-proxy variables can be <proxy>:port. I'll raise a separate issue for that.

Revision history for this message
Michael Foord (mfoord) wrote :

Changing code in proxyupdaterworker to unconditionally set environment variables (instead of just on first run as current logic is) allows me to add a charm to the environment (this failed before).

Revision history for this message
Michael Foord (mfoord) wrote :

I do still see an error from the deployed unit (charmrevisionworker) with the same problem - but the unit of the charm is in a started status. So there's obviously somewhere else with the same problem.

Revision history for this message
Michael Foord (mfoord) wrote :

Moving the proxyupdater worker to be started before other workers seems to remove the problem with the charmrevisionworker.

Revision history for this message
Michael Foord (mfoord) wrote :

Note: https://github.com/voidspace/juju/tree/charmstore-proxy

Builds and tests pass. Needs new tests (and more manual testing).

Revision history for this message
Michael Foord (mfoord) wrote :

Problem with charmrevisionworker still present (intermittently - so timing dependent it seems), but the changes seem good anyway. Adding tests.

Revision history for this message
Michael Foord (mfoord) wrote :

The discussed fix is landing now. https://github.com/juju/juju/pull/1528

It's still worth looking into why the "first" logic didn't work as expected as it *looks* correct. Also I'm continuing to investigate the charmrevision worker issue.

Revision history for this message
Adam Stokes (adam-stokes) wrote :

Thanks, is it possible to get this backported to 1.21, 1.22 as well?

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Yes Adam, I'll backport it to 1.21 and 1.22.

Changed in juju-core:
status: Triaged → Fix Committed
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Correction, so 1.21 does not use the same proxyupdater worker the same way, so it's not trivial to backport and we need to prioritize it if it's that important, but since we've already released 1.22 RC and 1.23 will be soon out I think backporting this should be a low priority.

no longer affects: juju-core/1.21
Revision history for this message
Michael Foord (mfoord) wrote :

The charmrevisionworker is started *after* the proxyupdater - so the fact that I *sometimes* see an error in the logs from the charmrevisionworker (unable to access the internet) is a simple race condition. Sometimes this worker is started before the proxy settings have been put in the environment. The right fix is for starting the proxyupdater to block until SetUp is completed.

The full call stack (traced in cmd/jujud/agent/machine.go - I probably need to confirm it's the same ordering in the unit agent) is:

MachineAgent

proxyupdater.New called from ->

postUpgradeAPIWorker
APIWorker
Run

charmrevisionworker created in ->

startEnvWorkers
StateWorker
stateStarter
newStateStarterWorker
Run

And APIWorker is called before newStateStarterWorker in MachineAgent.Run.

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

We won't fix this for 1.21 as the code changes between 1.21 and 1.22/1.23 are significant enough to make the effort moot in the face of 1.21.2 getting released (hopefully tomorrow) and the 1.22 release to follow shortly after (couple of weeks), which already has the fix. The only related fix we'll do for 1.21 is to improve the charmrevisionupdater worker's behavior - on download error it will retry after a few seconds, rather than after 24h as it does now. However, this is not guaranteed to resolve all charm download issues in restrictive environments using a proxy.

Revision history for this message
Michael Foord (mfoord) wrote :

The fix was backported to 1.22 incorrectly. Fixing now.

Revision history for this message
Adam Stokes (adam-stokes) wrote :

I tried 1.22-beta3 and was unable to provision any machines via local provider:

juju log: http://paste.ubuntu.com/10175762/
environments.yaml: http://paste.ubuntu.com/10175781/

I'll leave the machine up if you need anything else.

Revision history for this message
Adam Stokes (adam-stokes) wrote :

So just thinking out loud here, is it possible that during lxc create it is still having trouble getting the lxc image since it does a call out to wget? Are we passing any proxy information through to lxc during creation? What about for KVM? I am able to bootstrap our single installer which uses KVM intially, but then the containers within that environment fail to create/startup, just wondering if we're missing proxy settings there as well.

Revision history for this message
Michael Foord (mfoord) wrote :

I didn't look at container creation. It's highly likely that we don't pass proxy settings through to the container. wget *should* use proxy settings from the environment on the host machine though. It would be better to file "local provider fails behind a proxy" as a new bug though as this one was about charm download and is at "fix committed/released" stage.

Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Revision history for this message
Adam Stokes (adam-stokes) wrote :

FYI, I did get this to work finally, I didn't have all the required ips in the no-proxy setting.

Adam Stokes (adam-stokes)
Changed in cloud-installer:
status: Confirmed → Fix Committed
Adam Stokes (adam-stokes)
Changed in cloud-installer:
status: Fix Committed → Fix Released
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.