package-data-downloader hangs forever when attempting to download through cntlm proxy

Bug #1009436 reported by Adrianna Pińska on 2012-06-06
58
This bug affects 10 people
Affects Status Importance Assigned to Milestone
cntlm (Debian)
New
Unknown
cntlm (Ubuntu)
Medium
Unassigned
Precise
Undecided
Unassigned

Bug Description

[SRU] The debdiff attached to comment #22 backports cntlm-0.92.3-1ubuntu1 from Quantal to Precise.

It fixes the following bugs in Precise:
- package-data-downloader hangs forever when attempting to download through cntlm proxy (LP: #1009436)
- Can not play radio streams any more (LP: #659809)
- error when downloading files >2GB (LP: #1031670)
- cntlm does not work at reboot (LP: #825593)
- cntlm gpg error The following signatures were invalid: NODATA 2 (LP: #257210)

[IMPACT]
When behind a corporate proxy requiring NTLM authentication, users are unable to:
- install packages which download external files, e.g. flashplugin-installer and ttf-mscorefonts-installer (worked in Lucid)
- play internet radio streams (worked in Lucid)
- download files larger than 2GB in size
- download and install GPG keys through apt-get and apt-add-repository

[Test Cases]
Cntlm should be correctly configured and network proxy applied system wide as 127.0.0.1 port 3128 for HTTP, HTTPS and FTP, but not Socks.

- package-data-downloader hangs forever when attempting to download through cntlm proxy (LP: #1009436)
Run 'sudo apt-get install flashplugin-installer'
0.91 behaviour: flashplugin-installer downloads, installs, displays 'flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.236.orig.tar.gz' and then stops responding.
0.92 behaviour: flashplugin-installer downloads, installs and then downloads and installs the actual Flash plugin.

- Can not play radio streams any more (LP: #659809)
Open Rhythmbox and add http://shouthostdirect12.streams.bassdrive.com:8200/ as a New Internet Radio Station. Highlight the newly added station and click Play.
0.91 behaviour: no music is heard and a red No Entry icon appears next to the station.
0.92 behaviour: music starts playing.

- error when downloading files >2GB (LP: #1031670)
Run 'wget http://cdimage.ubuntu.com/edubuntu/dvd/current/quantal-dvd-amd64.iso'
0.91 behaviour: the message 'Connection closed at byte 0. Retrying' appears repeatedly.
0.92 behaviour: the download proceeds normally.

- cntlm does not work at reboot (LP: #825593)
Restart the computer with no network cable attached, open Firefox and enter a URL.
0.91 behaviour: the message 'The proxy server is refusing connections' appears.
0.92 behaviour: the message '502 connection timed out. cntlm proxy failed to complete the request' appears.

- cntlm gpg error The following signatures were invalid: NODATA 2 (LP: #257210)
Run 'sudo add-apt-repository ppa:ginggs/ppa', press Enter to continue when promtped.
0.91 behaviour: the message 'gpg: requesting key 08CC41D2 from hkp server keyserver.ubuntu.com' appears and then stops responding.
0.92 behaviour: the key is downloaded and installed normally.

[Regression Potential]
Minimal: cntlm has no dependants and no dependencies besides libc6.

I am proposing a backport instead of cherry-picking individual patches for Precise because of the difficulties I experienced in trying to cherry-pick r306. I found that the current commits relied on other changes that were not present in the two-year old version 0.91 in Precise. For example, cntlm with r306 broke in subtle ways (certain pages were not rendered correctly) when r281 was not included.
In addition, I found unrelated changes in the commits, for example, more detailed debug logging and dummy checks introduces to suppress compiler warnings, which were difficult to extract.
Lastly, while running the 0.91 version with debugging information on, cntlm would segfault every couple of days while under heavy load (several workstations sharing one cntlm gateway), whereas this did not occur in the 0.92 version from Quantal.

In short, I believe backporting the Quantal version will give us a more stable base, without the increased regression potential of cherry-picking multiple patches onto a two-year old version.

---------------------------------------------------------------------------------------

I previously reported this in #983559, but it seems that it is a completely unrelated issue.

1) Ubuntu 12.04 LTS
2) 0.119ubuntu8.4
3) I expected to be able to install packages which need to download external files (e.g. flashplugin-installer, ttf-mscorefonts-installer)
4) package-data-downloader is unable to download these files through my cntlm proxy

At university, I can only access the outside world through an NTLM proxy, so I direct everything through a cntlm proxy running locally on my machine. Everything used to work as expected, but since the change in Precise to the way that packages download external files, I am unable to install these packages unless I hack the package-data-downloader script to replace urllib with an external wget call (patch included).

This is definitely not an issue with finding the proxy settings when using sudo -- I have verified that the environment variables are found as expected. I have also reproduced this issue with a minimal python example which I have run as my normal user.

If I monitor the /tmp/ directory after issuing the command to install one of these packages, I can see that a temporary file is created and grows until it reaches the expected size of the file to be downloaded. However, the script does not detect that the file has finished downloading, and hangs forever, without giving any indication of an error, until it is terminated.

Since I am able to download these files without any difficulty using wget, I have patched my script to use wget instead of urllib. When I do this, everything works as expected. I therefore believe that there is a bug in urllib which is triggered by cntlm. I have looked for a known bug which could be responsible, but I haven't been able to find anything yet.

In the meantime, using wget instead of urllib fixes the problem. I don't know if this is considered a tidy solution, but it works.

Adrianna Pińska (confluence) wrote :

I can't seem to attach two attachments to one report, so here's my minimal urllib example. As an aside, setting the proxies from the environment explicitly seems to be completely unnecessary, since urllib should do that by default unless proxies are explicitly turned off.

I tried downloading different things to see if it made any difference. I haven't conducted an exhaustive search, but I have found that I can download the Google homepage, but not a specific image file. I don't know how the upstream NTLM proxy is configured exactly, so this could just be a coincidence.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in update-notifier (Ubuntu):
status: New → Confirmed
Adrianna Pińska (confluence) wrote :

I mangled the diff; here's the correct one.

Graham Inggs (ginggs) wrote :

I'm confirming that when using cntlm, package-data-downloader downloads the file but never installs it, no error messages either.
When using our corporate proxy, package-data-downloader downloads the file and installs it successfully.

I arranged with our proxy administrators for an authentication-bypass rule so that we can access http://archive.canonical.com/ through the proxy without requiring authentication.

The attachment "Replace urllib with wget in downloader script" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Adrianna Pińska (confluence) wrote :

Things that are missing from my patch: complete removal of the urllib code, and removal of files from /tmp/ once they are no longer needed. I would be happy to work to clean this up further if this approach is actually a viable long-term solution.

Pedro Pedruzzi (pedro-pedruzzi) wrote :

This is probably caused by the following cntlm bug:

http://sourceforge.net/tracker/?func=detail&aid=3512077&group_id=197861&atid=963162

It can be seen by the network traffic that package-data-downloader (through urllib) is using HTTP/1.0, the whole file is transfered and then the connection hangs. See the bug report for more details.

It is fixed in cntlm 0.93beta3 (supposedly).

As a workaround you could try to use proxychains chained "in front" of cntlm (untested).

Pedro Pedruzzi (pedro-pedruzzi) wrote :

I am targeting this to cntlm package because apparently there is nothing wrong with update-notifier nor with urllib.

affects: update-notifier (Ubuntu) → cntlm (Ubuntu)
Adrianna Pińska (confluence) wrote :

Interesting. I have always had apt-add-repository hang on fetching keys from the default server, and I wonder if that's related. It works if I use a different server. I think I've also had problems with easy_install and pip in the past.

I will test the proxy chain workaround, and I eagerly await an updated cntlm package.

Stefano Rivera (stefanor) wrote :

Can we find out exactly which commit fixes it?

I'll happily do a PPA build of 0.93beta3 next week, when I'm back from Croatia.

Graham Inggs (ginggs) wrote :

I believe this is revision 306:

Rev 306, 2012-04-07 02:54:51
Author: dave
Log message:
* Properly handle non-HTTP/1.1 keep-alive (many proxies respond with 1.1 even when client is 1.0)
and they happily add keep-alive connection, which we held open and 1.0 clients hanged. Now, HTTP
version is detected based purely on the client's original request and explicit keep-alive/close
headers are added to the replies based on its version. When "close" header is present, Cntlm
actually closes the connection to satisfy truly 0.9/1.0 clients (even though parent proxy
returned keep-alive) and now both 1.1 and 1.0 clients get what they expect. May need more testing,
but seems to work OK in both the DIRECT and PROXY handlers.
* Added more detailed debug logging of HTTP header versions from clients and servers/proxyies alike.
* Some additional write() return value checks (some dummy) to keep new GCC from spewing the
return-value-not-used warnings, even though we sometimes really don't care about the result.
Adding this specific warning suppression flag in the Makefile broke compilation with older GCC
versions.

Attaching output of:
$ svn di --old=http://svn.awk.cz/cntlm/trunk@305 --new=http://svn.awk.cz/cntlm/trunk@306 >r306.patch

Graham Inggs (ginggs) wrote :

I had a go at a PPA build. I ended up including r306 and r281 (some pages weren't rendering correctly without r281 as well).

It is available here:
https://launchpad.net/~ginggs/+archive/ppa

I have tested installing 'flashplugin-installer' and 'ttf-mscorefonts' packages as well as fetching keys with 'add-apt-repository'.

chugun (chugunv) wrote :

After installing cntlm from https://launchpad.net/~ginggs/+archive/ppa i get an error when try to install flashplugin-installer or ttf-mscorefonts-installer.

ttf-mscorefonts-installer: downloading http://downloads.sourceforge.net/corefonts/andale32.exe
Traceback (most recent call last):
  File "/usr/lib/update-notifier/package-data-downloader", line 234, in process_download_requests
    dest_file = urllib.urlretrieve(files[i])[0]
  File "/usr/lib/python2.7/urllib.py", line 93, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.7/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.7/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 351, in open_http
    'got a bad status line', None)
IOError: ('http protocol error', 0, 'got a bad status line', None)

Adrianna Pińska (confluence) wrote :

The PPA version of cntlm works for me.

Graham Inggs (ginggs) wrote :

@chugunv: I am unable to reproduce your problem. Does running cntlm with debugging information provide any clues?

$ sudo service cntlm stop
$ sudo cntlm -v

Changed in cntlm (Debian):
status: Unknown → New
Changed in cntlm (Ubuntu):
importance: Undecided → Medium
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cntlm - 0.92.3-1ubuntu1

---------------
cntlm (0.92.3-1ubuntu1) quantal; urgency=low

  * Cherry-pick r306 from 0.93 to properly handle non-HTTP/1.1 keep-alive
    (LP: #1009436)
 -- Graham Inggs <email address hidden> Wed, 01 Aug 2012 23:04:45 +0200

Changed in cntlm (Ubuntu):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cntlm (Ubuntu Precise):
status: New → Confirmed
Graham Inggs (ginggs) wrote :

[SRU] The attached debdiff backports cntlm-0.92.3-1ubuntu1 from Quantal to Precise.

It fixes the following bugs in Precise:
- package-data-downloader hangs forever when attempting to download through cntlm proxy (LP: #1009436)
- Can not play radio streams any more (LP: #659809)
- error when downloading files >2GB (LP: #1031670)
- cntlm does not work at reboot (LP: #825593)
- cntlm gpg error The following signatures were invalid: NODATA 2 (LP: #257210)

[IMPACT]
When behind a corporate proxy requiring NTLM authentication, users are unable to:
- install packages which download external files, e.g. flashplugin-installer and ttf-mscorefonts-installer (worked in Lucid)
- play internet radio streams (worked in Lucid)
- download files larger than 2GB in size
- download and install GPG keys through apt-get and apt-add-repository

[Regression Potential]
Minimal: cntlm has no dependants and no dependencies besides libc6

Graham Inggs (ginggs) on 2012-08-02
description: updated
Graham Inggs (ginggs) on 2012-08-04
description: updated
Graham Inggs (ginggs) on 2012-08-07
description: updated
Graham Inggs (ginggs) wrote :

Updated debdiff with changelog and references to 'win' and 'rpm' directories removed.

description: updated

Hello Adrianna, or anyone else affected,

Accepted cntlm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cntlm/0.92.3-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cntlm (Ubuntu Precise):
status: Confirmed → Fix Committed
tags: added: verification-needed
Graham Inggs (ginggs) wrote :

I have tested on both the i386 and amd64 versions of cntlm from precise-proposed.

$ sudo apt-get install flashplugin-installer
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ttf-bitstream-vera ttf-dejavu ttf-xfree86-nonfree xfs
The following NEW packages will be installed:
  flashplugin-installer
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/8 076 B of archives.
After this operation, 139 kB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously unselected package flashplugin-installer.
(Reading database ... 246356 files and directories currently installed.)
Unpacking flashplugin-installer (from .../flashplugin-installer_11.2.202.238ubuntu0.12.04.1_amd64.deb) ...
Processing triggers for update-notifier-common ...
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.238.orig.tar.gz
Installing from local file /tmp/tmpPXTuze.gz
Flash Plugin installed.
Setting up flashplugin-installer (11.2.202.238ubuntu0.12.04.1) ...

tags: added: verification-done
removed: verification-needed

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cntlm - 0.92.3-0ubuntu0.1

---------------
cntlm (0.92.3-0ubuntu0.1) precise-proposed; urgency=low

  * Backport 0.92.3-1ubuntu1 to Precise as an SRU (LP: #1009436):
    - Properly handle non-HTTP/1.1 keep-alive (LP: #1009436, #257210)
    - Support SHOUTcast (ICY) internet radio protocol (LP: #659809)
    - Fix error when downloading files >2GB (LP: #1031670)
    - Resolve proxy hostname on demand, not at startup (LP: #825593)

cntlm (0.92.3-1ubuntu1) quantal; urgency=low

  * Cherry-pick r306 from 0.93 to properly handle non-HTTP/1.1 keep-alive
    (LP: #1009436)

cntlm (0.92.3-1) unstable; urgency=low

  * New upstream release. Closes: #652725, #588920.
  * Fix Init script error, thanks Martijn. Closes: #588683.
  * Correct spellings in man page.
  * Update Standards Version, no changes needed.
 -- Graham Inggs <email address hidden> Wed, 29 Aug 2012 16:26:00 +0200

Changed in cntlm (Ubuntu Precise):
status: Fix Committed → Fix Released

Still not work on ubuntu my command:

export http_proxy=http://127.0.0.1:3128; sudo -E apt-get --reinstall install ttf-mscorefonts-installer

Give me a user and password request to use parent proxy (never required for already configured svn git and http)

ttf-mscorefonts-installer: downloading http://downloads.sourceforge.net/corefonts/andale32.exe
Enter username for Cntlm for parent at 127.0.0.1:3128:
Enter password for in Cntlm for parent at 127.0.0.1:3128:

-------------------------

cntlm:
  Installato: 0.92.3-0ubuntu0.1
  Candidato: 0.92.3-0ubuntu0.1
  Tabella versione:
 *** 0.92.3-0ubuntu0.1 0
        500 http://archive.ubuntu.com/ubuntu/ precise-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.91~rc6-0ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages

Forgot the ubuntu release into previous post

Description: Ubuntu 12.04.2 LTS
Release: 12.04
Codename: precise

Graham Inggs (ginggs) wrote :

@marcobra: You need to set the proxy in the Network Settings applet and click the 'Apply system wide' button.
You may have to log out and in again as well.

@Graham the proxy was already set into network properties....

Graham Inggs (ginggs) wrote :

@marcobra: Then you shouldn't need to export http_proxy.
Also, I notice your cntlm is requesting a username and password, can you try editing /etc/cntlm.conf and storing your username and password (or better yet, your NTLM hashes) there?

@graham i tried almost all already done all suggestions you are sending me...
- also not exporting http_proxy var BTW the system already export them as set in network properties:

here the: set | grep -i http result

http_proxy=http://127.0.0.1:3128/
https_proxy=https://127.0.0.1:3128/
....

- my network have squid as proxy with win user/domain ntlmV2 auth helper
- Storing it as plain text or/and hash found with -H /etc/cntlm.conf made the cntlm based proxy service not working i get user/passwd dialog request for all http request and also don't solve also the issue

Always i get ...

ttf-mscorefonts-installer: downloading http://downloads.sourceforge.net/corefonts/andale32.exe
Enter username for Cntlm for parent at 127.0.0.1:3128:
Enter password for in Cntlm for parent at 127.0.0.1:3128:

I think is some bug related to ttf-mscorefonts-installer internal instruction row related to Python-urllib/1.17 as i can read into ttf-mscorefonts-installer cntl proxy request debug log, i get 407 error the local proxy settings
Seems related to this http://stackoverflow.com/questions/1481398/python-urllib2-https-and-proxy-ntlm-authentication

Host => downloads.sourceforge.net
User-Agent => Python-urllib/1.17
cntlm[17662]: 127.0.0.1 GET http://downloads.sourceforge.net/corefonts/andale32.exe
NTLM Request:
    Domain: cda
  Hostname: ubuntu-desktop
     Flags: 0xA208B205

Sending PROXY auth request...
Host => downloads.sourceforge.net
User-Agent => Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0
Proxy-Authorization => NTLM TlRMTVNTUAABAAAABbIIogMAAwAuAAAADgAOACAAAABVQlVOVFUtREVTS1RPUENEQQ==
Content-Length => 0

Reading PROXY auth response...
HEAD: HTTP/1.0 407 Proxy Authentication Required
Server => squid
Date => Fri, 12 Apr 2013 09:06:02 GMT
Content-Type => text/html
Content-Length => 1418
Expires => Fri, 12 Apr 2013 09:06:02 GMT
X-Squid-Error => ERR_CACHE_ACCESS_DENIED 0
X-Cache => MISS from linuxbackup.urbanistica.it
X-Cache-Lookup => NONE from linuxbackup.urbanistica.it:3128
Connection => close
Discarding 1418 bytes.
cntlm[17662]: No Proxy-Authenticate, NTLM not supported?
Proxy closed on us, reconnect.
Sending headers (6)...

Dirty solved by doing this:

mkdir ~/mscorefonts~/mscorefonts
cp /usr/share/package-data-downloads/ttf-mscorefonts-installer ~/mscorefonts
chenged with editor the content add wget in front to the fonts urls wget http://... , removed the hash row;
run it with :
sh mscorefonts
to donwload the exe fonts with wget

get the font dir with pwd

then reconfigured the package, telling the font dirs
sudo dpkg-reconfigure ttf-mscorefonts-installer

All done...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.