url_downloadable ignores proxy settings

[Daniel Potter]

Binary package hint: update-manager

TEST CASE:
1. define a proxy server somewhere (gconf, apt.conf, synaptic)
2.use iptables to disallow access to port 80
3. run update-manager and click on the "upgrade" button
4. verify that the upgrade hangs

4. install update-manager from jaunty-proposed
5. repeat step 3
6. verify that it now continues

When updating to 9.10 from from 9.04, the update-manager uses a utility function called "UpdateManager.Core.utils.url_downloadable". This function skips urllib2 - which is properly configured to use the proxy settings - and instead directly interacts with httplib, which requires the caller to properly set up the request in order to use the proxy.

Changing the url_downloadable function to use the proxy resolves the issue.

Specifically, I after:

  if scheme == "http":
    import httplib
    try:

I added:

      proxy = os.getenv('http_proxy')
      if (proxy):
        path = scheme + netloc + path
        netloc = proxy

This resolved the problem. This, however, isn't really a fix - it relies on the fact that init_proxy sets the http_proxy environment variable and ignores anything else that might set up urllib2 to properly use a proxy server.

[Daniel Potter (dmpotter)]

I should probably have mentioned that the end result is that, if you're behind a firewall, after clicking the "Upgrade" button to start a distribution upgrade, the Update Manager appears to hang. The reason is that it's sitting in url_downloadable, waiting for the connection to time out.

Correcting the function so that it uses the proxy causes it to launch instantly.

[Muelli (ubuntu-bugs-auftrags-killer)]

Yeah, this patch totally fixes the issue for me. Now I'm finally able to upgrade. Thanks man.
Confirming, that update-manager freezes when trying to connect to the internet through the configured proxy.

[Michael Vogt (mvo)]

Thanks Daniel for the bug and the fix. I know its not a ideal fix, but it should be good enough (and most importantly minimal enough) for jaunty. I do some investigation now into a fix based on urrlib2 (it just need to be made to do a HEAD request on http and a size() rquest on ftp). But init_proxy() should deal with all the common cases for setting up http_proxy

[Michael Vogt (mvo)]

The last comment should end with "... for jaunty" (to keep the fix as minimal as possible for now).

[Martin Pitt (pitti)]

Accepted update-manager into jaunty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Henning Moll (drscott)]

Does this also fix the following problem with '/usr/bin/do-release-upgrade' which i have with jaunty?

The python script tries to download something like http://changelogs.ubuntu.com/meta-release, but fails because of not handling proxy authentification. The problematic code is in /usr/share/pyshared/UpdateManager/Core/MetaRelease.py around line 251. It generates
407: ('Proxy Authentication Required', 'You must authenticate with this proxy before proceeding.'),

[Michael Vogt (mvo)]

@Henning: what kind of proxy authentication is required? A simple username,password should be handled. Or do you need more elaborate mechanisms?

[Henning Moll (drscott)]

@Michael: Well, it is just simple authentication. Here's what i did:

$ sudo env | grep http_proxy; sudo /usr/bin/do-release-upgrade
[sudo] password for user:
http_proxy=http://user:<email address hidden>:3128/
Checking for a new ubuntu release
No new release found

If i modifiy the mentioned python script to print out the error code it says 407.

A "wget http://changelogs.ubuntu.com/meta-release" works without problems.

[Henning Moll (drscott)]

after downloading the content of http://changelogs.ubuntu.com/meta-release manually to /var/lib/update-manager/meta-release, the script runs a little bit further:

Debug output of do-release-upgrade now shows:

$ sudo DEBUG_UPDATE_MANAGER=1 /usr/bin/do-release-upgrade
Checking for a new ubuntu release
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: http://changelogs.ubuntu.com/meta-release
MetaRelease.download()
have self.metarelease_information
MetaRelease.parse()
current dist name: 'jaunty'
new dist: <UpdateManager.Core.MetaRelease.Dist object at 0xb78780ac>
trying to find suitable mirror
mirror_from_sources_list: jaunty
trying to find suitable mirror
mirror_from_sources_list: jaunty
Failed Upgrade tool signature
Failed Upgrade tool
Done downloading
Failed to fetch
Fetching the upgrade failed. There may be a network problem.

[Muelli (ubuntu-bugs-auftrags-killer)]

Heya,

On 30.10.2009 14:08, Henning Moll wrote:
> Well, it is just simple authentication.
Well, if urllib was used, it doesn't support Proxies with
authentication: http://docs.python.org/library/urllib.html#index-699

One has to patch it in manually, using e.g. this method:
http://bytes.com/topic/python/answers/22918-proxy-authentication-using-urllib2

[Michael Vogt (mvo)]

Hello Henning, could you please try the following patch:

=== modified file 'UpdateManager/Core/utils.py'
--- UpdateManager/Core/utils.py 2009-01-27 13:48:14 +0000
+++ UpdateManager/Core/utils.py 2009-10-30 11:05:50 +0000
@@ -72,6 +72,10 @@
   (scheme, netloc, path, querry, fragment) = urlparse.urlsplit(uri)
   if scheme == "http":
     import httplib
+ proxy = os.getenv("http_proxy")
+ if (proxy):
+ path = scheme + netloc + path
+ netloc = proxy
     try:
       c = httplib.HTTPConnection(netloc)
       c.request("HEAD", path)
@@ -115,10 +115,10 @@
   * then into gconf (if gconfclient was supplied)
   """
   SYNAPTIC_CONF_FILE = "/root/.synaptic/synaptic.conf"
- proxy = None
+ proxy = os.getenv("http_proxy")
   # generic apt config wins
   apt_pkg.InitConfig()
- if apt_pkg.Config.Find("Acquire::http::Proxy") != '':
+ if not proxy and apt_pkg.Config.Find("Acquire::http::Proxy") != '':
     proxy = apt_pkg.Config.Find("Acquire::http::Proxy")
   # then synaptic
   elif os.path.exists(SYNAPTIC_CONF_FILE):

and see if that helps?

[Michael Vogt (mvo)]

Henning, as a workaround (that should not need the second patch), does it help if you define the proxy and the username/password in the gnome proxy configuration ? Or is it there already?

[Henning Moll (drscott)]

@Muelli: yes, the code in /usr/share/pyshared/UpdateManager/Core/MetaRelease.py uses urllib. Well, it uses urllib2...

@Michael: Your patch fixes my problem in "update-manager". Thats great! But it did not fix it in 'do-release-upgrade'.

[Henning Moll (drscott)]

@Michael: my last comment was in reply to your first patch. Now i will try your second one...

[Henning Moll (drscott)]

Michael, the proxy configuration has been also in the gnome proxy configuration ever since and it does not work without the patch.
Your second patch again fixes "update-manager" but not "do-release-upgrade"

[Henning Moll (drscott)]

The proposed version does not work for me.

$ apt-cache policy update-manager
update-manager:
  Installiert: 1:0.111.10
  Kandidat: 1:0.111.10
  Versions-Tabelle:
 *** 1:0.111.10 0
        400 http://archive.ubuntu.com jaunty-proposed/main Packages
...

A click on "check" should get a fresh version of /var/lib/update-manager/meta-release. This fails silently. There is no such file in /var/lib/update-manager/meta-release afterwards

[Henning Moll (drscott)]

... The same applies to the non-root version /home/<user>/.update-manager-core/meta-release

[jdobry (jdobry)]

second patch again fixes "update-manager" but not "do-release-upgrade" and it is problem if somebody have not X installed in server

[jdobry (jdobry)]

this patch http://launchpadlibrarian.net/34695623/lala.diff works for "do-release-upgrade" correcly
I am not see, that patch is commited but not released, sorry.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:0.130

---------------
update-manager (1:0.130) lucid; urgency=low

  * UpdateManager/UpdateManager.py:
    - set heading for the release-upgrader download window
  * DistUpgrade/DistUpgradeController.py:
    - force lts for lucid cdrom upgrades
  * AutoUpgradeTester/profile/:
    - fix missing BaseMetaPackages
  * AutoUpgradeTester/UpgradeTestBackendQemu.py:
    - fix AddRepo code
  * DistUpgrade/DistUpgradeConfigParser.py:
    - fix getWithDefault() to use the correct get{int,float,boolean}
      function based on the type of the default (LP: #465619)
      Thanks to Brian Murray for spotting this bug
  * DistUpgrade/DistUpgradeController.py:
    - fix typo (LP: #470011)
  * updated for karmic->lucid upgrades and hardy->lucid upgrades
  * UpdateManager/Core/utils.py::
    - fix url_downloadable() when a proxy needs to be used
      (LP: #446552)

 -- Michael Vogt <email address hidden> Tue, 03 Nov 2009 14:07:20 +0100

[Henning Moll (drscott)]

A summary of my observations:

My situation:
Normal actions in update-manager (1:0.111.9) work. I am able to get normal package updates. So in this part, update-manger handles my auth-proxy correctly. Without the credentials, the proxy would not handle any requests. The credentials are stored in gconf. I always checked the correctness of environment variable http_proxy before running a command from command line.

The jaunty-proposed package 1:0.111.10 does not solve my problem. The notification "a new version of ubuntu..." is not shown.
The lucid-released package 1:0.130 (which i installed for testing on jaunty) does not solve my problem. Notification not shown.

Both of Michaels patches (applied on 1:0.111.9) solve my problem for the gui update-manager. But both did not help for do-release-update (this is in contrast to observations made by jdobry)

The reason for failing on do-release-update is, that the problem is - at least partially - independend of url_downloadable. On problem exists in /usr/share/pyshared/UpdateManager/Core/MetaRelease.py around line 251. The code generates
407: ('Proxy Authentication Required', 'You must authenticate with this proxy before proceeding.'), So i assume that the fix of the patch has to be applied on more places.

Should the bug be reopended for lucid?

[Renan (renanbr)]

i got the same problem, from 9.04 to 9.10
my network proxy settings was configured, but this ignored that

marcobra's work around worked perfect for me (from duplicate)
https://bugs.launchpad.net/ubuntu/+source/update-manager-core/+bug/440229/comments/4

[Michael Vogt (mvo)]

@Henning: I think we should open a different bug for the proxy authentication. I would love to polish the patch a bit and send you a new version for testing.

[Michael Vogt (mvo)]

I just got confirmation on irc that the problem is fixed for regular proxy users on irc. I would like to suggest to push it to update as we got no regression reports so far and solve the authentication proxy issue independently (to not slow the deployment of this fix).

[Henning Moll (drscott)]

Michael wrote:
>I think we should open a different bug for the proxy authentication.

Just created: #479391

[Karl Rolfe (karl-rolfe)]

I updated to the proposed version then applied the 2nd patch and it hasn't worked for me.
With the non patched version in my /var/squid/log/access.log the first entry after hitting the check button is:
1257825448.330 1 192.168.5.91 TCP_DENIED/407 1771 GET http://changelogs.ubuntu.com/meta-release - NONE/- text/html
  no credentials passed to the proxy - as we would expect given explanations in this conversation.

However with the patch applied, it seems that there is no attempt made to download changelogs.ubuntu.com/meta-release, the first line in the access.log is:
1257825577.334 669 192.168.5.91 TCP_REFRESH_HIT/304 194 GET http://archive.ubuntu.com/ubuntu/dists/jaunty/Release.gpg kgnews DIRECT/91.189.88.134 -

My user is kgnews and so credentials are passed to the proxy. However there is no call made to changelogs.ubuntu.com and I am not presented with the button to "Upgrade", but am simply told my system is up to date.

[Karl Rolfe (karl-rolfe)]

I should add that in the rest of the log after hitting the check button there are no calls to changelogs.ubuntu.com just several to archive.ubuntu.com which all are passed.

[Karl Rolfe (karl-rolfe)]

 ==> check that env contains http_proxy setting
administrator@ubuntu-box-1:~$ sudo sh -c export
.
. content deleted from brevity
.
export http_proxy='kgnews:#####@http://proxy.ourproxy.org.au:3128'
 ==> yes, env is set with username and password and pointing to the proxy
 ==> then run update
administrator@ubuntu-box-1:~$ sudo DEBUG_UPDATE_MANAGER=yes update-manager
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: http://changelogs.ubuntu.com/meta-release
MetaRelease.download()
NO self.metarelease_information
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: http://changelogs.ubuntu.com/meta-release
MetaRelease.download()
NO self.metarelease_information
/var/lib/python-support/python2.6/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
  super(Connection, self).__init__(*args, **kwargs)
administrator@ubuntu-box-1:~$

The result in Update Manager is "Your system is up-to-date" and no upgrade button is presented.
and a check of the Squid access.log shows that no connection attempt was made to http://changelogs.ubuntu.com
This is with Michael's second patch applied.

[Gunther Stengl (gunther-stengl)]

The proposed solution installing the version from "jaunty-proposed" works for me. I am not using proxy authentication but must use a proxy.
My usage: sudo-shell# http_proxy="11.22.33.44:80" do-release-upgrade
It does not seem to use the user's gnome-proxy-settings.
Thanks.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:0.111.10

---------------
update-manager (1:0.111.10) jaunty-proposed; urgency=low

  * UpdateManager/Core/utils.py:
    - fix url_downloadable() when a proxy needs to be used
      (LP: #446552)

 -- Michael Vogt <email address hidden> Wed, 29 Apr 2009 15:50:57 +0200

[Thomas E Jenkins (thomas-jenkins)]

FWIW here is a patch that rewrites url_downloadable to use urllib2 instead of pulling http_proxy out of the environment.

[Thomas E Jenkins (thomas-jenkins)]

Here's the same path now against the right file and less a typo.

[Thomas E Jenkins (thomas-jenkins)]

I guess I wasn't testing what I thought I was testing. Here is the patch again less another typo. Sorry for the noise.

[Martin Pitt (pitti)]

Michael, can you please check and upload if you deem it appropriate? Thanks!

[Serge van Ginderachter (svg)]

It seems this bug still persists in Karmic, trying to upgrade to Lucid beta. Shouldn't this be fixed urgently before the Lucid release?

[Markus Strobl (mstrobl2)]

Confirm the problem still exists in Karmic. I applied the original patch from the bug report and after that update-manager -d works.

[Torsten Hilbrich (torsten-hilbrich-secunet)]

I also can confirm the bug to be present in update-manager 1:0.126.9 of karmic. Neither the proxy setting in synaptic nor the environment variable http_proxy are consulted when trying to the update.

The patch in use-urllib2-in-url_downloadable-3.diff fixed the problem for me.

Quite a long-standing bug, I had to apply the same patch when updating from jaunty to karmic :-(

[Riku Voipio (riku-voipio)]

karmic -> lucid path is still broken, since only jaunty and lucid versions were "fixed".

Also, the *proper* fix is to use urllib2 instead of that manual code in current lucid version. The "urllib2 doesn' support proxy auth" is poor excuse. urrlib2 should be fixed in python instead of working it around in applications using python... Not doing it properly gives a poor user experience as some python apps work with proxy, some not, others work with auth proxy and some not...

[ttp (human-ttp)]

"better patch" lala.diff fixed the problem for me again. I'm having the same problem when tried to upgrade to karmic.

[jstammi (jstammi)]

At least for the karmic -> lucid path the upgrade does work indeed - but only with waiting for ~2 hours :-(. After this time of waiting the upgrade works as expected (with using the proxy), see #574439.

[Michael Vogt (mvo)]

I uploaded a karmic version of the jaunty fix to karmic-proposed now.

[Martin Pitt (pitti)]

Accepted update-manager into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[hideki (hideki)]

update-manager from karmic-proposed works fine for me.

Thank you.

[Ville Ranki (ville-ranki)]

Martin's package solved the problem for me too.

[Henning Moll (drscott)]

It works - even with proxy authentication. bug #479391 seems to be a duplicate now.

[Marco De Paolini (marco-depaolini)]

Just upgraded to update-manager-core 1:0.126.10 and do-release-upgrade still not work for me on karmic amd64.

$ sudo env | grep http_proxy;sudo DEBUG_UPDATE_MANAGER=yes do-release-upgrade
http_proxy=http://xxxx:yyyy@192.168.64.20:3128
Checking for a new ubuntu release
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: http://changelogs.ubuntu.com/meta-release
MetaRelease.download()
NO self.metarelease_information
No new release found

Am I missing something?

[Guenter Roeck (public-roeck-us)]

update-manager from karmic-proposed fixed the problem for me.

Guenter

[Mike (mike-fdb)]

I'm second to comment 47.
do-release-upgrade from update-manager-core 1:0.126.10 doesn't work for me
I *do not* use authentification on proxy.

root@ws027:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=9.10
DISTRIB_CODENAME=karmic
DISTRIB_DESCRIPTION="Ubuntu 9.10"

root@ws027:~# uname -a
Linux ws027 2.6.31-21-generic #59-Ubuntu SMP Wed Mar 24 07:28:56 UTC 2010 i686 GNU/Linux

root@ws027:~# aptitude show update-manager-core
Пакет: update-manager-core
Состояние: установлен
Автоматически установлен: нет
Версия: 1:0.126.10
...

root@ws027:~# env|grep http_proxy
http_proxy=http://192.168.60.7:3222/

root@ws027:~# DEBUG_UPDATE_MANAGER=yes do-release-upgrade
Checking for a new ubuntu release
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: http://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: http://changelogs.ubuntu.com/meta-release
MetaRelease.download()
NO self.metarelease_information
No new release found

[Mike (mike-fdb)]

Please ignore my previous comment, there was error in my proxy settings!
Sorry for bugspam.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:0.126.10

---------------
update-manager (1:0.126.10) karmic-proposed; urgency=low

  * UpdateManager/Core/utils.py:
    - fix proxy detection in url_downloadable (LP: #446552)
  * UpdateManager/Core/DistUpgradeFetcherCore.py:
    - add missing "logging" import (LP: #475941)
 -- Michael Vogt <email address hidden> Thu, 06 May 2010 16:13:48 +0200

[Nikita (nikita-maizy)]

I have the same problem when upgrading from 10.04 to 10.10.
I'm behind proxy with basic auth (squid), proxy settings sets in env and apt.cont. Installing and updating packages works well.

> sudo do-release-upgrade
do nothing, so I replaced /var/lib/update-manager/meta-release from http://changelogs.ubuntu.com/meta-release manualy, then all works fine.

[Nikita (nikita-maizy)]

I'm also have problem with proxy, becouse I have separate proxy for internet connection and for apt (accepts only package downloading), but init_proxy() in utils.py prefer Acquire::http::Proxy then env settings. So I can't connect to http://changelogs.ubuntu.com from second proxy.

Solution for me was temporary set apt proxy the same as internet proxy.

[Daniel (hackie)]

Just trying to update from natty (11.04) to oneiric (11.10) and having the same problem: do-release-upgrade, started from console, hangs. http_proxy set, but Proxy settings in gnome are not set. Any console application should use http_proxy as first option