curl and pycurl is not compiled with sftp support

[Alex]

Binary package hint: curl

Running curl --version, I see:
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps
SFTP is obviously missing, although it is supported by curl upstream
The SFTP is also missing from python-pycurl, probably because libcurl doesn't have it.

[Martin Lindhe (martinlindhe)]

I second that.
Installed php5-curl to use libcurl with php scripts to read/write to a sftp, only to find that the packaged curl is built without sftp support.
Please enable sftp protocol support in curl!

[Brian Murray (brian-murray)]

sftp and scp support are unavailable with curl version 7.19.5-1ubuntu2 in Karmic Koala.

[Martin Lindhe (martinlindhe)]

To generate a curl with ssh support on Ubuntu 9.04:

sudo apt-get install build-essential debhelper libssh2-1-dev
sudo apt-get source libcurl3
sudo apt-get build-dep libcurl3

cd curl-7.18.2/debian

gedit rules

   find and replace "--without-libssh2" with "--with-libssh2"

cd ..

sudo dpkg-buildpackage

cd ..

sudo dpkg -i curl_7.18.2-8ubuntu4.1_amd64.deb
sudo dpkg -i libcurl3_7.18.2-8ubuntu4.1_amd64.deb
sudo dpkg -i libcurl3-gnutls_7.18.2-8ubuntu4.1_amd64.deb

[Martin Lindhe (martinlindhe)]

Upon further inspection, it appears that the debian package is configured with "--without-libssh2"

Can someone shed some light as to why libssh2 is disabled?
It makes curl unable to handle sftp or scp protocols.

[Rusty Shackleford (rshackleford-deactivatedaccount)]

It sure would be nice if the package could be built with "--with-libssh2" for Lucid's release. Any chance of that happening?

[goraxe (goraxe)]

I have attached a patch for this. One of our applications depends on this, we currently are building our own version of this package with patch applied, I have just had to rebuild for lucid.

[Martin Lindhe (martinlindhe)]

goraxe: your patch contains unrelated changes.
All that is needed is in curl/debian/rules

- cd debian/build-gnutls && ./configure ${CONFIGURE_ARGS} --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ssl --with-gnutls --without-libssh2
+ cd debian/build-gnutls && ./configure ${CONFIGURE_ARGS} --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ssl --with-gnutls --with-libssh2

/martin

[Martin Lindhe (martinlindhe)]

created a minimal patch to debian/rules to fix the issue

As a side note I have been using this to enable ssh in curl on a production server since September 2009 and it has been working perfectly.

[Nigel Babu (nigelbabu)]

This patch has been reviewed as part of operation cleansweep. Thanks for your patch. This particular change is inherited from debian, could you please open a bug in Debian and attach the patch? It would be much better to hear upstream Debian thoughts on this.

[goraxe (goraxe)]

I have opened a bug with the debian bug tracking system

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587735

[Cameron Smith (cameron-smith)]

Here are Martin Lindhe's instructions from post #3 above, updated for 10.04, again assuming you are using 64-bit Ubuntu. The resulting curl installation works just as the curl mapage says it should, against an OpenSSL server.

==========
#the line below will create a temporary directory and move you to it - so that you can easily clean up the build files later
# you are short on disk space
mkdir -p ~/temp/curl
cd ~/temp/curl

sudo apt-get install build-essential debhelper libssh2-1-dev
sudo apt-get source libcurl3
sudo apt-get build-dep libcurl3

cd curl-7.19.7/debian

gedit rules

   find and replace "--without-libssh2" with "--with-libssh2"

cd ..

sudo dpkg-buildpackage

cd ..

sudo dpkg -i curl_7.19.7-1ubuntu1_amd64.deb
sudo dpkg -i libcurl3_7.19.7-1ubuntu1_amd64.deb
sudo dpkg -i libcurl3-gnutls_7.19.7-1ubuntu1_amd64.deb

#at this point you may move away from, and then remove the entire ~/temp/curl directory if you wish
# alternately, you may choose just to keep the 3 .deb files which were the end product of the building,
# in case you have to use them again in the future.

[Sebastien Bacher (seb128)]

is that still an issue with the current version?

[Martin Lindhe (martinlindhe)]

according to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587735#15
the latest unstable debian packages (7.21.0-1) has sftp enabled

[Andreas Olsson (andol)]

If I understand the debian/rules file correctly, the --without-libssh2 option should only affect libcurl3-gnutls and not libcurl3? Futher it seems like the command line curl uses the (openssl) libcurl3?

The problem with curl and libcurl3 seems more related to a specific Ubuntu delta. That is that Ubuntu drops the build-dependency on libssh2-1-dev. When I rebuilt curl 7.21.0-1ubuntu1 (current maverick) with libssh2-1-dev readded as a build dependency sftp support worked just fine.

[Martin Lindhe (martinlindhe)]

Andreas: I think you are confusing ssh and ssl/tls support.

[Andreas Olsson (andol)]

Martin: How so?

[Martin Lindhe (martinlindhe)]

Andreas wrote "If I understand the debian/rules file correctly, the --without-libssh2 option should only affect libcurl3-gnutls and not libcurl3?"

gnutls is "Transport Layer Security, a network protocol and successor to Secure Sockets Layer (SSL)", it is unrelated to ssh.

Andreas wrote "Futher it seems like the command line curl uses the (openssl) libcurl3?"

Correct. however this bug is about missing sftp (ssh) support and not about missing ssl support (it is indeed there).

Andreas wrote "The problem with curl and libcurl3 seems more related to a specific Ubuntu delta. That is that Ubuntu drops the build-dependency on libssh2-1-dev."

According to mine and others previously analysis the rules/debian file are from Debian repo who also have disabled ssh support. See related upstream bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587735

Andreas wrote "When I rebuilt curl 7.21.0-1ubuntu1 (current maverick) with libssh2-1-dev readded as a build dependency sftp support worked just fine."

How so? the rules/debian file shipped has a "--without-libssh2" configure paramater which seems to be the cause of the error.

[Andreas Olsson (andol)]

No, I am not confusing ssh with ssl/tls. Yet, it is very possible that I am not expressing myself very clearly.

The reason I am mentioning gnutls and openssl is to differentiate between the libraries/packages libcurl3 and libcurl3-gnutls.

The option --without-libssh2 is only used when building the libcurl3-gnutls package. Hence it has nothing to do with the ability to use sftp:// with anything which uses/depends on libcurl3, such as curl or php5-curl.

Whatever libcurl3 can provide sftp support or not depends on whatever libssh2-1-dev is present during compile time.

I am not saying that --with(out)-libssh2 doesn't matter. There are a multitude of tools (including python-pycurl) depending on libcurl3-gnutls. My point is that that compile option isn't the whole stole.

[Martin Lindhe (martinlindhe)]

Andreas:

You are indeed correct.

I've myself recompiled and only installed the resulting libcurl to get SSH support (both command line and in php5-curl).
Before doing this, i also installed the libssh2-1-dev.

This would result in a working setup but my conclusions as to why were obviously flawed (ps. thanks for pointing this out!)

Attached is a corrected patch against curl-7.21.0 (maverick).

I added the dependency against the "curl" package. However I'm not sure if it should rather be against "libcurl3".

Can someone help confirm this solution so we can try to push a patch upstream.

[Andreas Olsson (andol)]

I am afraid it is not as simple as re-adding the build-dependency on libssh2-1-dev. It was not removed by mistake, but as an explicit decision, per bug #175891.

For curl to be able to depend on libssh2 it too would have to be in Main. See https://wiki.ubuntu.com/UbuntuMainInclusionRequirements for more information.

[Martin Lindhe (martinlindhe)]

Andreas, thanks for researching this.

I don't know how to proceed with resolving this issue since I dont have enough experience with these things.

As far as I can see there are two options

a) open a new bug report requesting libssh2 to be moved to main

or

b) create a separate package in universe which would enable ssh support in curl if installed.

I think method A would be easier to do, however it may be a more politicial issue than a technical one to resolve.

As for method B, i wouldn't know how we could do that

[Andreas Olsson (andol)]

Being more of a triager than a developer myself I am not sure myself what the best route is. Assuming you are comfortable with IRC you might want to try #ubuntu-devel or #ubuntu-motu on freenode.

[Martin Lindhe (martinlindhe)]

I have opened a MIR request in lp bug #681423

[Steel-Cat (panther-toppoint)]

Hello Martin,

your MIR was closed for inactivity... what a pity...

Is the maintainer of libssh2 not able to resolve the comments from Kees Cook?

How we can advance?

Steel-Cat

[KennethOnah (onah-kenneth)]

This is 2015 and still curl does not support sftp out of the box. Why so?

[Martin (ub71-martin)]

Yes, why ? Please ?

[Uqbar (uqbar)]

This is actually 2016.
15.10 curl is:

    [Uqbar@Feynman ~] curl --version
    curl 7.43.0 (x86_64-pc-linux-gnu) libcurl/7.43.0 GnuTLS/3.3.15 zlib/1.2.8 libidn/1.28 librtmp/2.3
    Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
   Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

Moreover the upstream bug got fixed long ago: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587735 .

What else?

[mdyn (tamerlaha-gmail)]

7 years old bug... niiice.

[Uqbar (uqbar)]

As of 16.04 we're still lacking behind: NO SFTP!!!

[Uqbar@Feynman ~] curl --version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

C'mon, guys! Is it that difficult to pull from Debian?
May I say it's a shame?

[Hugo Gonzalez (hugonz)]

Beginning 2018, want to upload to a webhost (no one supports plain FTP anymore) and I hit this. Please, can this be put in before the next LTS? Now it's the time.

[Valentin Wittich (vwittich)]

One year later (2019-02) the LTS (18.04.1, bionic) still has that issue!

Why is that excption in the debian build:

# cat curl3-7.58.0/debian/rules | grep -A 3 Ubuntu
ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
 CONFIGURE_ARGS += --without-libssh2
endif

[Valentin Wittich (vwittich)]

Here is how to build curl with libssl support for Ubuntu 18.04. LTS:

sudo apt-get install build-essential debhelper libssh-dev
sudo apt-get source curl
sudo apt-get build-dep curl

cd curl-*

sudo patch < ~/Downloads/ubuntu_libssl.patch

sudo dpkg-buildpackage -uc -us
# -us Do not sign the source package.
# -uc Do not sign the .changes file.

cd ..

sudo dpkg -i curl_*.deb
sudo dpkg -i libcurl3-*.deb
sudo dpkg -i libcurl3-gnutls_*.deb

sudo apt-mark hold curl
sudo apt-mark hold libcurl3
sudo apt-mark hold libcurl3-gnutls
# sudo apt-mark unhold <package-name>

Hope that helps somebody.

[Hugo Gonzalez (hugonz)]

El jue., 31 de ene. de 2019 07:34, Valentin Wittich <
<email address hidden>> escribió:

> Here is how to build curl with libssl support for Ubuntu 18.04. LTS:
>

It helps me, thanks!

>
>

[Sebastien Bacher (seb128)]

curl has a libssh (!= libssh2) backend now and that library is in main, which means we can finally enable the ssftp backend!
https://github.com/curl/curl/commit/c92d2e1

[Sebastien Bacher (seb128)]

Debian request to change from libssh2 to libssh is on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897950

[Sebastien Bacher (seb128)]

Could someone describe a test case using the backend which we could use to test that the feature is working if we enable it to bionic (the 'curl --version' output tells us that the backend is enabled but it would be good to also check that it's actually correctly working)

[Launchpad Janitor (janitor)]

This bug was fixed in the package curl - 7.63.0-1ubuntu1

---------------
curl (7.63.0-1ubuntu1) disco; urgency=medium

  * debian/control, debian/rules:
    - build with libssh instead of libssh2, that's a better maintained
      library and it's in Ubuntu main (lp: #311029)

 -- Sebastien Bacher <email address hidden> Thu, 31 Jan 2019 15:29:39 +0100

[Valentin Wittich (vwittich)]

> Could someone describe a test case using the backend which we could use to
> test that the feature is working if we enable it to bionic (the 'curl
> --version' output tells us that the backend is enabled but it would be
> good to also check that it's actually correctly working)

Not sure what you mean by test case using the backend. But I compiled version 7.63.0 and it looks like it works find. Though I just used git-ftp to upload via sftp, but that worked smooth.

Here is what I did to install it with ubuntu 18.04. I had to compile debhelper too.

git clone https://git.launchpad.net/ubuntu/+source/debhelper
cd debhelper/
sudo apt-get install build-essential po4a
dpkg-buildpackage -uc -us
cd ..
sudo apt-get install dwz
sudo dpkg -i debhelper_12ubuntu1_all.deb

git clone https://git.launchpad.net/ubuntu/+source/curl
cd curl/
sudo apt-get install build-essential libssh-dev
dpkg-buildpackage -b -rfakeroot -us -uc
cd ..
sudo dpkg -i libcurl*.deb
sudo dpkg -i curl*.deb

Here the versions output:

# curl --version
curl 7.63.0 (x86_64-pc-linux-gnu) libcurl/7.63.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) libssh/0.7.0/openssl/zlib nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-12-12
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

[Alexander Blinne (sunday)]

Please, please, please backport this fix to bionic!

[Valentin Wittich (vwittich)]

One prob I found is that uploading via IPv4 is crazy slow... though I don't understand if it has to do with the switch from libssl2 to libssl...

With a quick search I found an old blog entry (https://daniel.haxx.se/blog/2014/05/14/why-sftp-is-still-slow-in-curl/) which discribes some probs with SFTP.

Not sure if that is an issue...