bzr crashed with error in _curl_perform(): (28, 'SSL connection timeout at 298225') connecting to Launchpad

[Roland Dreier]

Binary package hint: bzr

Running

$ bzr get https://code.launchpad.net/~ubuntu-core-dev/ubuntu/natty/udev/ubuntu

gives me a big traceback and doesn't work. Not sure if the problem is on the launchpad side or not, but even if it is, bzr should probably just print an error message rather than crashing.

ProblemType: Crash
DistroRelease: Ubuntu 11.04
Package: bzr 2.3.0-2
ProcVersionSignature: Ubuntu 2.6.38-6.34-generic 2.6.38-rc7
Uname: Linux 2.6.38-6-generic x86_64
Architecture: amd64
BzrDebugFlags: set()
BzrPlugins:
 bash_completion /usr/lib/pymodules/python2.7/bzrlib/plugins/bash_completion [2.3.0]
   bzrtools /usr/lib/python2.7/dist-packages/bzrlib/plugins/bzrtools [2.3.1]
   launchpad /usr/lib/pymodules/python2.7/bzrlib/plugins/launchpad [2.3.0]
   netrc_credential_store /usr/lib/pymodules/python2.7/bzrlib/plugins/netrc_credential_store [2.3.0]
   news_merge /usr/lib/pymodules/python2.7/bzrlib/plugins/news_merge [2.3.0]
BzrVersion: 2.3.0
CommandLine:
 ['/usr/bin/bzr',
  'get',
  'https://code.launchpad.net/~ubuntu-core-dev/ubuntu/natty/udev/ubuntu']
CrashDb: bzr
Date: Wed Mar 16 09:30:55 2011
EcryptfsInUse: Yes
ExecutablePath: /usr/bin/bzr
FileSystemEncoding: UTF-8
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Alpha amd64 (20101206)
InterpreterPath: /usr/bin/python2.7
Locale: en_US.UTF-8
Platform: Linux-2.6.38-6-generic-x86_64-with-Ubuntu-11.04-natty
ProcCmdline: /usr/bin/python /usr/bin/bzr get https://code.launchpad.net/~ubuntu-core-dev/ubuntu/natty/udev/ubuntu
ProcEnviron:
 SHELL=/bin/bash
 PATH=(custom, user)
 LANG=en_US.UTF-8
 LANGUAGE=en_US:en
PythonVersion: 2.7.1
SourcePackage: bzr
Title: bzr crashed with error in _curl_perform(): (28, 'SSL connection timeout at 298225')
UpgradeStatus: Upgraded to natty on 2011-03-09 (6 days ago)
UserEncoding: UTF-8
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/bzrlib/commands.py", line 923, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/bzrlib/commands.py", line 1123, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/pymodules/python2.7/bzrlib/commands.py", line 691, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/pymodules/python2.7/bzrlib/commands.py", line 710, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/pymodules/python2.7/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/bzrlib/builtins.py", line 1183, in run
    from_location)
  File "/usr/lib/pymodules/python2.7/bzrlib/bzrdir.py", line 806, in open_tree_or_branch
    bzrdir = klass.open(location)
  File "/usr/lib/pymodules/python2.7/bzrlib/bzrdir.py", line 717, in open
    return BzrDir.open_from_transport(t, _unsupported=_unsupported)
  File "/usr/lib/pymodules/python2.7/bzrlib/bzrdir.py", line 747, in open_from_transport
    redirected)
  File "/usr/lib/pymodules/python2.7/bzrlib/lazy_import.py", line 128, in __call__
    return obj(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/bzrlib/transport/__init__.py", line 1670, in do_catching_redirections
    return action(transport)
  File "/usr/lib/pymodules/python2.7/bzrlib/bzrdir.py", line 734, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/pymodules/python2.7/bzrlib/controldir.py", line 750, in find_format
    return prober.probe_transport(transport)
  File "/usr/lib/pymodules/python2.7/bzrlib/bzrdir.py", line 1501, in probe_transport
    server_version = medium.protocol_version()
  File "/usr/lib/pymodules/python2.7/bzrlib/smart/medium.py", line 632, in protocol_version
    client_protocol.query_version()
  File "/usr/lib/pymodules/python2.7/bzrlib/smart/protocol.py", line 791, in query_version
    self.call('hello')
  File "/usr/lib/pymodules/python2.7/bzrlib/smart/protocol.py", line 631, in call
    self._request.finished_writing()
  File "/usr/lib/pymodules/python2.7/bzrlib/smart/medium.py", line 439, in finished_writing
    self._finished_writing()
  File "/usr/lib/pymodules/python2.7/bzrlib/transport/http/__init__.py", line 651, in _finished_writing
    data = self._medium.send_http_smart_request(self._buffer)
  File "/usr/lib/pymodules/python2.7/bzrlib/transport/http/__init__.py", line 615, in send_http_smart_request
    code, body_filelike = t._post(bytes)
  File "/usr/lib/pymodules/python2.7/bzrlib/transport/http/_pycurl.py", line 273, in _post
    'Content-Type: application/octet-stream'])
  File "/usr/lib/pymodules/python2.7/bzrlib/transport/http/_pycurl.py", line 375, in _curl_perform
    curl.perform()
error: (28, 'SSL connection timeout at 298225')

[Martin Pool (mbp)]

bzr should not give a traceback for a network error.

As far as the error itself:

1- some users may be firewalled or need a proxy; in that case we'd expect it to fail consistently
2- bzr or a library may be doing something strange that makes failures more likely (perhaps the timeout is unreasonably short?)
3- Launchpad may not be accepting the connection fast enough, and therefore it is a real timeout that they could potentially fix

[Robert Collins (lifeless)]

This is talking to the primary launchpad website (code.launchpad.net) - we definitely don't timeout there; I suspect a firewall or some such, and this bug should focus on better diagnostics (and no traceback :)). Marking the Launchpad task invalid as such.

[Cameron Hart (bitshifternz)]

When I got it my Internet connection was on the fritz and kept dropping out.

[Chris (ccouzens)]

I just got the same error. My Internet connection is decent. I'm on a university Internet connection. While IT services have done weird things with the connection in the past, I have no reason to suspect it at the moment.

I triggered it with "bzr launchpad-login ccouzens" but it is the same error. I can't get it to error a 2nd time.

$ bzr launchpad-login ccouzens
bzr: ERROR: pycurl.error: (28, 'SSL connection timeout at 298545')

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 926, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1126, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 691, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 713, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/launchpad/__init__.py", line 278, in run
    account.check_lp_login(name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/launchpad/account.py", line 106, in check_lp_login
    data = _transport.get_bytes('~%s/+sshkeys' % username)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/__init__.py", line 610, in get_bytes
    f = self.get(relpath)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/__init__.py", line 126, in get
    code, response_file = self._get(relpath, None)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_pycurl.py", line 171, in _get
    return self._get_full(relpath)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_pycurl.py", line 206, in _get_full
    self._curl_perform(curl, header)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_pycurl.py", line 375, in _curl_perform
    curl.perform()
error: (28, 'SSL connection timeout at 298545')

You can report this problem to Bazaar's developers by running
    apport-bug /var/crash/bzr.1000.2011-05-24T20:17.crash
if a bug-reporting window does not automatically appear.

[Martin Pool (mbp)]

@chris so just to confirm, you got this only once and can't reproduce it?

[Robert Collins (lifeless)]

Sidnei pointed me at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594150
and
http://comments.gmane.org/gmane.comp.web.curl.library/31904

[Sidnei da Silva (sidnei)]

The fix was comitted upstream:
  https://github.com/bagder/curl/commit/d4e000906ac4ef243258a5c9a819a7cde247d16a

We might need to get this into an SRU (for maverick and later iirc).

[Michael Vogt (mvo)]

I uploaded the fix to natty-proposed. If you can reproduce, please give feedback once the version for natty-proposed hits the archive.

It does not apply cleanly to maverick, would be nice if someone could have a look.

[Martin Pool (mbp)]

We'll need to at least take the preceding patch[1], and perhaps more

[1] https://github.com/bagder/curl/commit/cbf4961bf3e

[Martin Pitt (pitti)]

Please fix in oneiric ASAP, so that this can progress into -updates.

[Martin Pitt (pitti)]

Accepted curl into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Sidnei da Silva (sidnei)]

No sign of the package in natty-proposed yet, so I guess there's a backlog in the buildd.

[Martin Pitt (pitti)]

proposed package failed to build, setting back to "in progress"

[Steve Beattie (sbeattie)]

The reason the proposed package FTBFS is because the curl package is a bit quirky and non-standard in the way it uses quilt in order to build against gnutls and nss in the same build. Attached is an updated debdiff (based off of the current version in natty-updates) that includes the cherrypicked upstream patch in a place in the series where it won't break the build. It would be great if someone could sponsor this.

[Steve Beattie (sbeattie)]

(Oh, I also adjusted the patch headers to be DEP-3 compliant. It's verified to build to completion in a local natty-amd64 chroot.)

[Steve Beattie (sbeattie)]

Also, in my local builds, there were no regressions in the upstream curl testsuite as compared to the build of curl_7.21.3-1ubuntu1.2; the relevant parts from umt compare-log are below (the testsuite gets run multiple times, one for each build against the different crypto libraries):

@@ -4773,7 +4780,7 @@
 test 2003...FAILED
 TESTDONE: 502 tests out of 515 reported OK: 97%
 TESTFAIL: These test cases failed: 283 284 285 286 1007 1009 1013 1049 1093 1094 1099 2002 2003<
-TESTDONE: 586 tests were considered during 4530 seconds.
+TESTDONE: 586 tests were considered during 4523 seconds.
 TESTINFO: 71 tests were skipped due to these restraints:
 TESTINFO: "no stunnel" 25 times (300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 400, 401, 403, 404, 406, 407, 408, 409, 560, 1097, 1112)
 TESTINFO: "curl lacks scp support" 10 times (601, 603, 605, 607, 617, 619, 621, 623, 629, 631)

@@ -5725,7 +5732,7 @@
 test 2003...FAILED
 TESTDONE: 502 tests out of 515 reported OK: 97%
 TESTFAIL: These test cases failed: 283 284 285 286 1007 1009 1013 1049 1093 1094 1099 2002 2003<
-TESTDONE: 586 tests were considered during 4522 seconds.
+TESTDONE: 586 tests were considered during 4538 seconds.
 TESTINFO: 71 tests were skipped due to these restraints:
 TESTINFO: "no stunnel" 23 times (300, 301, 302, 303, 304, 305, 306, 309, 310, 311, 312, 313, 400, 401, 403, 404, 406, 407, 408, 409, 560, 1097, 1112)
 TESTINFO: "curl lacks scp support" 10 times (601, 603, 605, 607, 617, 619, 621, 623, 629, 631)

@@ -6678,7 +6685,7 @@
 test 2003...FAILED
 TESTDONE: 502 tests out of 515 reported OK: 97%
 TESTFAIL: These test cases failed: 283 284 285 286 1007 1009 1013 1049 1093 1094 1099 2002 2003<
-TESTDONE: 586 tests were considered during 4530 seconds.
+TESTDONE: 586 tests were considered during 4533 seconds.
 TESTINFO: 71 tests were skipped due to these restraints:
 TESTINFO: "no stunnel" 23 times (300, 301, 302, 303, 304, 305, 306, 309, 310, 311, 312, 313, 400, 401, 403, 404, 406, 407, 408, 409, 560, 1097, 1112)
 TESTINFO: "curl lacks scp support" 10 times (601, 603, 605, 607, 617, 619, 621, 623, 629, 631)

[Jamie Strandboge (jdstrand)]

ACK the natty-proposed patch. FYI, inspecting oneiric's code, curl still doesn't have the fix. I'll prepare an upload.

[Jamie Strandboge (jdstrand)]

Uploading 7.21.3-1ubuntu1.3 to natty-proposed. Unsubscribing ubuntu-sponsors. If someone provides a patch for maverick, feel free to resubscribe.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curl - 7.21.6-3ubuntu2

---------------
curl (7.21.6-3ubuntu2) oneiric; urgency=low

  * debian/patches/timeout_bug_736216: cherry pick upstream
    git revision d4e000906ac4ef243258a5c9a819a7cde247d16a to fix
    handshake timeout bug (LP: #736216). Thanks to Sidnei da Silva
    and Michael Vogt
 -- Jamie Strandboge <email address hidden> Wed, 13 Jul 2011 12:08:54 -0500

[Clint Byrum (clint-fewbar)]

Hello Roland, or anyone else affected,

Accepted curl into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Sidnei da Silva (sidnei)]

A quick test seems to indicate that updating to libcurl3-gnutls=7.21.3-1ubuntu1.3 fixes my problems with landscape-client, bzr and apt-transport-https. I'll leave landscape-client running for some 24h and check the logs to make sure.

[Sidnei da Silva (sidnei)]

After installing the updated libcurl3-gnutls package the error hasn't happened over a period of 7h, where before it would happen at least once in a period of 30 minutes. I consider this bug fixed.

[C de-Avillez (hggdh2)]

Tagging verification-done-natty, based on Sidnei's previous comment.

[Benjamin Kerensa (bkerensa)]

Bug is occurring on Xubuntu 11.04 and its not per connection :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package curl - 7.21.3-1ubuntu1.3

---------------
curl (7.21.3-1ubuntu1.3) natty-proposed; urgency=low

  * debian/patches/timeout_bug_736216: cherry pick upstream
    git revision d4e000906ac4ef243258a5c9a819a7cde247d16a to fix
    handshake timeout bug (LP: #736216). Thanks to Sidnei da Silva
    and Michael Vogt
 -- Steve Beattie <email address hidden> Fri, 24 Jun 2011 11:36:02 -0700

[Colin Watson (cjwatson)]

Unsubscribing ubuntu-archive; I can't quite make out why we were subscribed in the first place, but there doesn't appear to be an archive admin action for us to perform right now. Feel free to resubscribe us with an explanation if this changes or I'm just missing something.

[Rolf Leggewie (r0lf)]

maverick has seen the end of its life and is no longer receiving any updates. Marking the maverick task for this ticket as "Won't Fix".