GnuTLS recv error (-9): A TLS packet with unexpected length was received

[TJ]

On Precise 12.04 whilst attempting:

GIT_CURL_VERBOSE=1 git clone -v https://git01.codeplex.com/typescript

the operation fails after the final git pack-file has been received and the already-created repository is deleted from the file system.

...
> POST /typescript/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2.433.g9808ce0.dirty
Host: git01.codeplex.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 611

* upload completely sent off: 611out of 611 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Pragma: no-cache
< Content-Type: application/x-git-upload-pack-result
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
< Date: Thu, 31 Jan 2013 21:43:55 GMT
< Connection: close
<
remote: Counting objects: 149766, done.
remote: Compressing objects: 100% (10580/10580), done.
* GnuTLS recv error (-9): A TLS packet with unexpected length was received.
* Closing connection #0
remote: Total 149766 (delta 138201), reused 149559 (delta 138077)
Receiving objects: 100% (149766/149766), 198.98 MiB | 361 KiB/s, done.
error: RPC failed; result=56, HTTP code = 200
Resolving deltas: 100% (138201/138201), done.

git exits at this point but it deletes the entire cloned ./typescript directory.

I tried building the latest git binary and included an additional debug option in "http.c" that allowed me to set the protocol version using an environment option:

CURLOPT_SSLVERSION=1 git clone ...

where 1 = TLSv1, 2 = SSLv2, 3 = SSLv3.

I tried each protocol but the result was the same.

The knock-on bug here is that git ought not to delete what it has fetched - in this case more than 250MB of data.

I did try to build the latest gnutls but it needs a very recent version of libnettle which has the "rsa_decrypt_tr" function. I stopped at that point since I don't want to get into dependency and library version issues.

[TJ (tj)]

I've added the other packages that git depends upon since this might need to be dealt with higher up than libgnutls.

git < libcurl3-gnutls < libgnutls26

[Anders Kaseorg (andersk)]

Fascinating. I can reproduce, but I recommend you take this to the upstream Git mailing list, since they will more likely be able to help you figure out what’s going wrong.

http://git-scm.com/community

[TJ (tj)]

This is a gnutls issue; it could affect any application that makes use of it.

I've already mentioned it on the git developers mailing list and it has been seen once or twice before affecting git.

Additional research seems to indicate this is a known intentional gnutls behaviour (that has been modified in very recent gnutls that makes use of a recent libnettle - as mentioned above). The issue is, apparently, the random size padding of packets to prevent communications compromise for stream ciphers.

Unfortunately the changes required are far too invasive for an SRU so we'll have to make do with a work-around.

I installed stunnel4 (which depends on openssl rather than gnutls) and created a reverse-proxy (client in stunnel terminology):

$ cat /etc/stunnel/rp-codeplex.com.conf
client = yes

[http]
accept = 8888
connect = git01.codeplex.com:443
TIMEOUTclose = 0

$ sudo sed -i 's/\(ENABLED\).*/\1=1/' /etc/default/stunnel4
$ sudo service stunnel4 restart

$ GIT_CURL_VERBOSE=1 git clone -v http://localhost:8888/typescript

...
> POST http://localhost:8888/typescript/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2.433.g9808ce0.dirty
Host: localhost:8888
Accept-Encoding: gzip
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 611

* upload completely sent off: 611out of 611 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Pragma: no-cache
< Content-Type: application/x-git-upload-pack-result
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
< Date: Thu, 31 Jan 2013 23:38:19 GMT
< Connection: close
<
remote: Counting objects: 149798, done.
remote: Compressing objects: 100% (10612/10612), done.
remote: Total 149798 (delta 138221), reused 149558 (delta 138077)
* Closing connection #0
Receiving objects: 100% (149798/149798), 198.99 MiB | 640 KiB/s, done.
Resolving deltas: 100% (138221/138221), done.
Checking out files: 100% (2851/2851), done.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Teräslilja (m-matti-a-lehtonen)]

With "GIT push", I got
> * Done waiting for 100-continue
> < HTTP/1.1 100 Continue
> * GnuTLS recv error (-9): A TLS packet with unexpected length was received.
> * Closing connection 6
> PUT XXX failed, aborting (56/100)

Didn't fix after upgrading (from 13.04) to and recompiling following libraries
nettle-2.7.1
gnutls-3.2.3
curl-7.32.0
git version 1.8.3.4

[Mirco Bauer (meebey)]

This also failed for me like this:

git clone https://git01.codeplex.com/katanaproject
Cloning into 'katanaproject'...
remote: Counting objects: 21605, done.
remote: Compressing objects: 100% (3386/3386), done.
remote: Total 21605 (delta 17488), reused 21605 (delta 17488)
Receiving objects: 100% (21605/21605), 6.47 MiB | 913.00 KiB/s, done.
error: RPC failed; result=56, HTTP code = 200
Resolving deltas: 100% (17488/17488), done.

I could workaround this by doing:
mkdir katanaproject
cd katanaproject
git init
git remote add -f origin https://git01.codeplex.com/katanaproject
git fetch
git checkout -B master origin/master

[Giuseppe Pasquino (giuseppe-pasquino)]

This issue is still present in Ubuntu 14.04 LTS and git version 1.9.1

[Wolverine (nareshtechs)]

To clarify things, no the issue is not within GIT itself. The issue comes from libcurl3-openssl. Most of the GIT binaries that are distributed (either from GIT PPA or Ubuntu) is compiled to work with libcurl3-openssl.

Hence, it's necessary to recompile the GIT from source and link it to libcurl4-openssl. The issue can only be solved after Ubuntu or Git PPA is compiled against libcurl4-openssl instead of libcurl3-openssl.

[Jonathan Elchison (jelchison)]

Thanks, Wolverine.

I've confirmed that the workaround posted at http://askubuntu.com/a/187199 succeeds in the following environment:
* git 1.9.1-1
* Ubuntu 14.04 LTS on i686

For the record, the workaround I'm posting recompiles git with libcurl4-openssl-dev (instead of libcurl4-gnutls-dev).

[thulle (thulle)]

I think this bug also affects ngIRCd.
Connecting to a SSL-enabled ngIRCd using gnutls-cli fails on;
*** Fatal error: A TLS packet with unexpected length was received.
Connecting with openssl s_client fails with;
140615615354528:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:

Recompiling ngIRCd against libgnutls28-dev fixes the problem.

[Anders Kaseorg (andersk)]

Unfortunately, Git compiled with OpenSSL cannot be distributed for licensing reasons.

http://lintian.debian.org/tags/possible-gpl-code-linked-with-openssl.html
https://people.gnome.org/~markmc/openssl-and-the-gpl.html

So we will need to find some other solution.

[Vindicator (vindicator)]

I just encountered this issue with Ubuntu 15.04 64-bit while trying to clone nuget: https://nuget.codeplex.com/workitem/8339

[Jeff (liujinfengjeff)]

This also failed for me like this:
cd /home/jeff/jef
git clone https://github.com/torch/distro.git ~/torch --recursive

error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
fatal: clone of 'https://github.com/torch/cutorch.git' into submodule path 'extra/cutorch' failed

last,I could workaround this by doing:
cd /home/jeff
git clone https://github.com/torch/distro.git ~/torch --recursive

and fix it. Sometimes because of network failure.I hope my answer can help you.

[Jeff (liujinfengjeff)]

Sometimes because of network failure,try more than once maybe you can fix it.

[Joe Damato (q-joe)]

This bug in gnutls also affects users of apt-transport-https which is used by APT for accessing repositories via TLS. Fixing this bug would be a great help for SSL-enabled APT repositories.

[Julian Andres Klode (juliank)]

Nothing to be done for apt here due to legal reasons. I'm marking this as invalid, but might just remove the task completely later on

[David Ayers (ayers)]

TJ stated in early 2013:

"Additional research seems to indicate this is a known intentional gnutls behaviour (that has been modified in very recent gnutls that makes use of a recent libnettle - as mentioned above). The issue is, apparently, the random size padding of packets to prevent communications compromise for stream ciphers.

Unfortunately the changes required are far too invasive for an SRU so we'll have to make do with a work-around."

Where do these changes need be implemented? In GNUTLS?
Has anyone with sufficient understanding opened an informed report upstream at https://gitlab.com/gnutls/gnutls/issues that could be referenced here?
If not, would someone be willing / capable to do that?

Thanks!

[Lakshmi Kannan (lakshmi-9)]

We are hitting this error roughly 1% of the times when running our CI.

Err:1 https://packagecloud.io/StackStorm/staging-enterprise-unstable/ubuntu xenial/main amd64 st2flow amd64 2.2dev-1
  GnuTLS recv error (-54): Error in the pull function.

[Adam Voss (vossad01)]

This issue has shown up on StackOverflow. The advice there is to rebuild git :(

http://stackoverflow.com/q/38378914/1072626

I doubt it will be useful but here is the verbose output from my failure: https://gist.github.com/vossad01/cb65180966f63eeb10c24b8ebf8957b8

[Adam Voss (vossad01)]

My case ended up not being this bug. I recompiled with OpenSSL (devopscube.com/gnutls-handshake-failed-aws-codecommit/) and then got the following error instead:

Writing objects: 100% (1103/1103), 1.09 MiB | 0 bytes/s, done.
Total 1103 (delta 653), reused 1001 (delta 606)
error: RPC failed; curl 56 SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

I am guessing it was because I was pushing so many commits (upstream changes). I deleted my fork on GitHub and recreated it so that it would be up-to-date, then had no problem pushing my changes.

[Joe Damato (q-joe)]

I've changed the status for APT back to "Confirmed" - I've seen many cases of this bug reported. I don't follow the comment about legal reasons (#17). Ubuntu already ships apt-transport-https, which links against gnutls and suffers from this bug; no new code causing any new legal issues would be pulled in. Fixing this bug in gnutls would also fix it in apt-transport-https, so IMO, APT is affected by this bug just the same as git and curl.

[Julian Andres Klode (juliank)]

While this does affect apt, there is no actionable item here, as we can't switch to OpenSSL for legal reasons, and won't switch to another more obscure TLS library. Fixing the GnuTLS bug is dealt with by the gnutls task.

Curl and Git are not my topic.

[Julian Andres Klode (juliank)]

That said, curl already has a built against openssl, so the problem is "solved" there. The gnutls variant is of course affected, but that's for GnuTLS to fix.

[Joe Damato (q-joe)]

OpenSSL was recently relicensed: https://www.openssl.org/blog/blog/2017/03/22/license/. I'm not a lawyer, but it would be nice to get one to weigh in on how this affects the legal issues mentioned.

[Julian Andres Klode (juliank)]

They are in the process of relicensing - asking for permissions. This might go on for some years or might never conclude.

[Adrian Wilkins (adrian-wilkins)]

Also seeing this with libgnutls30 and git 2.17.1-ubuntu0 on 18.04