bzr update takes ages over FTP

[Ernst]

I have a remote repository with format 2a. With bzr 1.17, I tried to update my checkout. the command 'bzr update' does not seem to finish, on Windows and Linux. The message in my terminal is:

>bzr update
[#########\ ] Fetching revisions:Get stream source

I've attached the corresponding log of (a still running instance of) bzr.

[Ernst (ernst-blaauw)]

Well, it seemed bzr-gtk was causing some troubles. Still, bzr update is very, very slow (it is running for more than 20 minutes, I think). See the log file. Is this caused by the new format 2a? The repository is accesed over a ftp connection, that could be related?
For sure, with the 1.14 format the update was much, much faster.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ernst wrote:
> Well, it seemed bzr-gtk was causing some troubles. Still, bzr update is very, very slow (it is running for more than 20 minutes, I think). See the log file. Is this caused by the new format 2a? The repository is accesed over a ftp connection, that could be related?
> For sure, with the 1.14 format the update was much, much faster.
>
> ** Attachment added: "bzrlog2.txt"
> http://launchpadlibrarian.net/29488667/bzrlog2.txt
>

So it is 100% related to the fact that we are going over FTP, which doesn't
allow us to do a "partial download". So instead every time we need XX bytes
from the file, we download the whole content, extract those bytes and then
throw it away.

Potentially our FTP layer could be taught how to do some caching, and this
would probably improve dramatically. The main issue is how do you decide on a
lifetime for the cache, do you cache in memory or on disk, or ....

If you used bzr+ssh or sftp or http your results would most likely be
dramatically better.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKaf9ZJdeBCYSNAAMRAu1nAKCqyMsyoTZWFXXw49udvPCwGL5/lQCgtPpA
yNW4EdmI/i+qRt6t4qljbhY=
=ntnM
-----END PGP SIGNATURE-----

[Ernst (ernst-blaauw)]

Thanks for your answer :-)
For me, this means:
1) http only provides read only access (so no good solution)
2) I only have ssh access (not sftp) and cannot start bzr server

This means, atm there is no solution for me?

I'm trying to get sftp working, but I did not succeed yet.

On Fri, Jul 24, 2009 at 20:37, John A Meinel <email address hidden> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ernst wrote:
> > Well, it seemed bzr-gtk was causing some troubles. Still, bzr update is
> very, very slow (it is running for more than 20 minutes, I think). See the
> log file. Is this caused by the new format 2a? The repository is accesed
> over a ftp connection, that could be related?
> > For sure, with the 1.14 format the update was much, much faster.
> >
> > ** Attachment added: "bzrlog2.txt"
> > http://launchpadlibrarian.net/29488667/bzrlog2.txt
> >
>
> So it is 100% related to the fact that we are going over FTP, which doesn't
> allow us to do a "partial download". So instead every time we need XX bytes
> from the file, we download the whole content, extract those bytes and then
> throw it away.
>
> Potentially our FTP layer could be taught how to do some caching, and this
> would probably improve dramatically. The main issue is how do you decide on
> a
> lifetime for the cache, do you cache in memory or on disk, or ....
>
> If you used bzr+ssh or sftp or http your results would most likely be
> dramatically better.
>
> John
> =:->
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFKaf9ZJdeBCYSNAAMRAu1nAKCqyMsyoTZWFXXw49udvPCwGL5/lQCgtPpA
> yNW4EdmI/i+qRt6t4qljbhY=
> =ntnM
> -----END PGP SIGNATURE-----
>
> ** Summary changed:
>
> - bzr update takes ages
> + bzr update takes ages over FTP
>
> ** Changed in: bzr
> Importance: Undecided => Medium
>
> ** Changed in: bzr
> Status: New => Triaged
>
> --
> bzr update takes ages over FTP
> https://bugs.launchpad.net/bugs/404047
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ernst wrote:
> Thanks for your answer :-)
> For me, this means:
> 1) http only provides read only access (so no good solution)
> 2) I only have ssh access (not sftp) and cannot start bzr server
>
> This means, atm there is no solution for me?

Well, ftp *works* it is just slow.

>
> I'm trying to get sftp working, but I did not succeed yet.
>

I'm a bit surprised here. Perhaps you were expecting to be able to use a
relative url? For example:

 bzr log sftp://myhost/home/jameinel/myproject

versus potentially (but wouldn't work):
 bzr log myhost:myproject
 bzr log sftp://myhost/myproject

(though bzr log sftp://myhost/~/myproject should work)

In general, does an sftp client work? Or is there just no sftp support on the
server?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKaivEJdeBCYSNAAMRAmktAKDYNhk0cr0ITcGAA/PAWLmuk9fPqwCglV7C
ENuphQShgk2KaKdSSPLXrDw=
=dD4t
-----END PGP SIGNATURE-----

[Ernst (ernst-blaauw)]

No, there is no sftp support: I'm running DropBear, which is a ssh server
with low memory usage. Therefore, it is suitable for NAS or media devices.
However, it has not built-in support for sftp (although it seems it can be
added with a hack, but I did not succeed).
Bottom line: no sftp support ;-).

On Fri, Jul 24, 2009 at 23:46, John A Meinel <email address hidden> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ernst wrote:
> > Thanks for your answer :-)
> > For me, this means:
> > 1) http only provides read only access (so no good solution)
> > 2) I only have ssh access (not sftp) and cannot start bzr server
> >
> > This means, atm there is no solution for me?
>
> Well, ftp *works* it is just slow.
>
> >
> > I'm trying to get sftp working, but I did not succeed yet.
> >
>
>
> I'm a bit surprised here. Perhaps you were expecting to be able to use a
> relative url? For example:
>
> bzr log sftp://myhost/home/jameinel/myproject
>
> versus potentially (but wouldn't work):
> bzr log myhost:myproject
> bzr log sftp://myhost/myproject
>
> (though bzr log sftp://myhost/~/myproject should work)
>
> In general, does an sftp client work? Or is there just no sftp support on
> the
> server?
>
> John
> =:->
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFKaivEJdeBCYSNAAMRAmktAKDYNhk0cr0ITcGAA/PAWLmuk9fPqwCglV7C
> ENuphQShgk2KaKdSSPLXrDw=
> =dD4t
> -----END PGP SIGNATURE-----
>
> --
> bzr update takes ages over FTP
> https://bugs.launchpad.net/bugs/404047
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Martin Pool (mbp)]

It might get a bit tedious but you could pull over http and push over ftp.

[Ernst (ernst-blaauw)]

I have no clue how to set up this (is it possible to bind to different
checkout and commit servers?).
Besides, isn't this a huge security risk, as http is available for everyone?

On Sat, Jul 25, 2009 at 00:07, Martin Pool <email address hidden> wrote:

> It might get a bit tedious but you could pull over http and push over
> ftp.
>
> --
> bzr update takes ages over FTP
> https://bugs.launchpad.net/bugs/404047
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Ernst (ernst-blaauw)]

And isn't it possible to add ssh:// to the dumb servers and make it like
sftp? Or does sftp provide certain operatios which are not possible in ssh?
This could be useful to some users, as devices like my Networked Media Tank
become more and more popular and DropBear is the most suitable SSH solution
on those devices (because of the low resource usage). As DropBear does not
provide sftp, you have to use ftp which can be slow. A more efficient ssh
access would be welcome in such case.

On Sat, Jul 25, 2009 at 00:29, Ernst Blaauw <email address hidden> wrote:

> I have no clue how to set up this (is it possible to bind to different
> checkout and commit servers?).
> Besides, isn't this a huge security risk, as http is available for
> everyone?
>
>
> On Sat, Jul 25, 2009 at 00:07, Martin Pool <email address hidden> wrote:
>
>> It might get a bit tedious but you could pull over http and push over
>> ftp.
>>
>> --
>> bzr update takes ages over FTP
>> https://bugs.launchpad.net/bugs/404047
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>
>

[Martin Pool (mbp)]

2009/7/26 Ernst <email address hidden>:
> And isn't it possible to add ssh:// to the dumb servers and make it like
> sftp? Or does sftp provide certain operatios which are not possible in ssh?
> This could be useful to some users, as devices like my Networked Media Tank
> become more and more popular and DropBear is the most suitable SSH solution
> on those devices (because of the low resource usage). As DropBear does not
> provide sftp, you have to use ftp which can be slow. A more efficient ssh
> access would be welcome in such case.

If you can install bzr on the server and run bzr+ssh:// that will
probably be faster than either sftp or ftp. That will require Python
on the server.

If you're asking can we do a dumb transport over ssh then that's sftp.
 There's no point reinventing it. Note that sftp can be run as a
subprocess of sshd so there's no resource usage until it's used.

--
Martin <http://launchpad.net/~mbp/>

[Ernst (ernst-blaauw)]

Well, I'm trying to get bzr running on my 'server' (which is actually a
media device). See my bug report https://bugs.launchpad.net/bugs/404333.

Also, I did not succeed in enabling sftp on my media device, but I hope I
will succeed some day :-).

Thanks for your support! I got really fast answers on my bug reports /
questions.
Ernst

On Sun, Jul 26, 2009 at 02:13, Martin Pool <email address hidden> wrote:

> 2009/7/26 Ernst <email address hidden>:
> > And isn't it possible to add ssh:// to the dumb servers and make it like
> > sftp? Or does sftp provide certain operatios which are not possible in
> ssh?
> > This could be useful to some users, as devices like my Networked Media
> Tank
> > become more and more popular and DropBear is the most suitable SSH
> solution
> > on those devices (because of the low resource usage). As DropBear does
> not
> > provide sftp, you have to use ftp which can be slow. A more efficient ssh
> > access would be welcome in such case.
>
> If you can install bzr on the server and run bzr+ssh:// that will
> probably be faster than either sftp or ftp. That will require Python
> on the server.
>
> If you're asking can we do a dumb transport over ssh then that's sftp.
> There's no point reinventing it. Note that sftp can be run as a
> subprocess of sshd so there's no resource usage until it's used.
>
> --
> Martin <http://launchpad.net/~mbp/ <http://launchpad.net/%7Embp/>>
>
> --
> bzr update takes ages over FTP
> https://bugs.launchpad.net/bugs/404047
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Robert Collins (lifeless)]

On Fri, 2009-07-24 at 18:37 +0000, John A Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ernst wrote:
> > Well, it seemed bzr-gtk was causing some troubles. Still, bzr update is very, very slow (it is running for more than 20 minutes, I think). See the log file. Is this caused by the new format 2a? The repository is accesed over a ftp connection, that could be related?
> > For sure, with the 1.14 format the update was much, much faster.
> >
> > ** Attachment added: "bzrlog2.txt"
> > http://launchpadlibrarian.net/29488667/bzrlog2.txt
> >
>
> So it is 100% related to the fact that we are going over FTP, which doesn't
> allow us to do a "partial download". So instead every time we need XX bytes
> from the file, we download the whole content, extract those bytes and then
> throw it away.

I think this is exacerbated by the currently open bugs in pulling from
2a formats - we make many many times as many readv requests as we used
to; and while FTP does do some ugly stuff when doing readv at all, it
should be tolerable once those performance bugs are fixed.

-Rob