error on ftp servers that allow only one connection per client ip

[Timmie]

FTP temporary error: 421 Too many connections (2) from this IP. Retrying.
bzr: ERROR: ftplib.error_temp: 421 Too many connections (2) from this IP

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 893, in run_bzr_catch_errors
  File "bzrlib\commands.pyo", line 839, in run_bzr
  File "bzrlib\commands.pyo", line 539, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 2502, in run
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\workingtree_4.pyo", line 224, in commit
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\mutabletree.pyo", line 194, in commit
  File "bzrlib\branch.pyo", line 156, in _get_nick
  File "bzrlib\decorators.pyo", line 138, in read_locked
  File "bzrlib\branch.pyo", line 1990, in get_master_branch
  File "bzrlib\branch.pyo", line 123, in open
  File "bzrlib\bzrdir.pyo", line 778, in open
  File "bzrlib\bzrdir.pyo", line 806, in open_from_transport
  File "bzrlib\transport\__init__.pyo", line 1663, in do_catching_redirections
  File "bzrlib\bzrdir.pyo", line 793, in find_format
  File "bzrlib\bzrdir.pyo", line 1624, in find_format
  File "bzrlib\bzrdir.pyo", line 1634, in probe_transport
  File "bzrlib\transport\ftp\__init__.pyo", line 264, in get
  File "bzrlib\transport\ftp\__init__.pyo", line 160, in _reconnect
  File "bzrlib\transport\ftp\__init__.pyo", line 141, in _create_connection
  File "ftplib.pyo", line 132, in connect
  File "ftplib.pyo", line 214, in getresp
error_temp: 421 Too many connections (2) from this IP

bzr 1.11 on python 2.5.2 (win32)
arguments: ['bzr', 'commit']
encoding: 'cp1252', fsenc: 'mbcs', lang: 'de'
plugins:
  bzrtools C:\Programme\Bazaar\plugins\bzrtools [1.11]
  difftools C:\Programme\Bazaar\plugins\difftools [0.91]
  extmerge C:\Programme\Bazaar\plugins\extmerge [unknown]
  launchpad C:\Programme\Bazaar\plugins\launchpad [unknown]
  merge_into C:\Programme\Bazaar\plugins\merge_into [unknown]
  netrc_credential_store C:\Programme\Bazaar\plugins\netrc_credential_store [unknown]
  push_and_update C:\Programme\Bazaar\plugins\push_and_update [0.2dev]
  qbzr C:\Programme\Bazaar\plugins\qbzr [0.9.6]
  svn C:\Programme\Bazaar\plugins\svn [0.4.17]
  upload C:\Programme\Bazaar\plugins\upload [1.0dev]
  xmloutput C:\Programme\Bazaar\plugins\xmloutput [0.8.3]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Note: bzr from commandline, bzr was the only program using FTP at that moment.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim wrote:
> Public bug reported:
>
> FTP temporary error: 421 Too many connections (2) from this IP. Retrying.
> bzr: ERROR: ftplib.error_temp: 421 Too many connections (2) from this IP
>

...

> Note: bzr from commandline, bzr was the only program using FTP at that moment.
>
> ** Affects: bzr
> Importance: Undecided
> Status: New

I'm not quite sure when we are trying to connect twice to an ftp server,
but it might be an 'active' versus 'passive' ftp issue.

IIRC, 'active' ftp tells the server to connect back to the client in
order to transfer data, but tends to fail with stuff like NAT, etc. On
the other hand, I believe there is a way to transfer FTP data on the
same connection that you are issuing commands, and that may be
independent of active/passive.

Anyway, you could try using "aftp://" which sets the 'active' flag.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknvWosACgkQJdeBCYSNAAOORACfZfZ78jXP6L8jTl48KYSwhVnN
gjkAoIkjilDKR68fHixi/JtRapIAlwqN
=shoD
-----END PGP SIGNATURE-----

[Timmie (timmie)]

This still happens with version nr. 1.14

[Timmie (timmie)]

Hello,
for me this is now getting critical as I cannot commit to the remote repository.

I got this via PM:
>>>>> "Tim" == Tim Michelsen <email address hidden> writes:

     Tim> Hello,
     Tim> can you please help me here:
     Tim> ftp lib error: too many connections
     Tim> https://bugs.launchpad.net/bzr/+bug/365160

     Tim> Why does this happen?
     Tim> I am getting this quite often.

Do you get it 'often' or 'always' ?

If the former, the corresponding .bzr.log ('bzr version' will tell you where it resides) will help understand the problem.

     Tim> A restart of the computer doesn't migtigate.

Strange, sounds more like 'always' then...

     Tim> It occurs on:
     Tim> windows XP
     Tim> Bazaar 1.14
     Tim> using on commandline
     Tim> The FTP is on the local network (corporate)

You may check with your admins whether the server is keeping open connections even when bzr have closed them...

Be aware that running GUIs (or even TortoiseBzr) may also initiate connections to the remote branch (do you use checkouts or bound branches ?).

Basically, bzr shouldn't use more than one connection per branch and most operations should involve only one remote branch, so only one connection should be used at a given time.

I did a test:
create a temporary direcory and then:
bzr co ftp://myurl
=> right away it gives: too many connections...

If I go into the working copy I am developing in and do
bzr commit I get:
bzr commit
'module' object has no attribute 'O_NONBLOCK'
Unable to load plugin u'bzr_touch' from u'C:/Programme/Bazaar/plugins'
bzr: ERROR: exceptions.AttributeError: 'str' object has no attribute 'get'

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 727, in exception_to_return_code
  File "bzrlib\commands.pyo", line 922, in run_bzr
  File "bzrlib\commands.pyo", line 559, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 2878, in run
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\workingtree_4.pyo", line 226, in commit
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\mutabletree.pyo", line 194, in commit
  File "bzrlib\branch.pyo", line 208, in _get_nick
  File "bzrlib\decorators.pyo", line 138, in read_locked
  File "bzrlib\branch.pyo", line 2280, in get_master_branch
  File "bzrlib\branch.pyo", line 135, in open
  File "bzrlib\bzrdir.pyo", line 1679, in open_branch
  File "bzrlib\branch.pyo", line 1603, in open
  File "bzrlib\bzrdir.pyo", line 649, in find_repository
  File "bzrlib\bzrdir.pyo", line 625, in _find_containing
  File "bzrlib\bzrdir.pyo", line 883, in open_containing_from_transport
  File "bzrlib\bzrdir.pyo", line 837, in open_from_transport
  File "bzrlib\transport\__init__.pyo", line 1657, in do_catching_redirections
  File "bzrlib\bzrdir.pyo", line 824, in find_format
  File "bzrlib\bzrdir.pyo", line 1751, in find_format
  File "C:/Programme/Bazaar/plugins\svn\format.py", line 109, in probe_transport
  File "C:/Programme/Bazaar/plugins\svn\transport.py", line 89, in get_svn_ra_transport
  File "C:/Programme/Bazaar/plugins\svn\errors.py", line 130, in convert
  File "C:/Programme/Bazaar/plugins\svn\transport.py", line 260...

[Timmie (timmie)]

This is an BZR error. With SVN I do not have these problems.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim wrote:
> This is an BZR error. With SVN I do not have these problems.
>

I'm pretty sure SVN doesn't work over ftp, so I don't see how that is
relevant.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoAWOAACgkQJdeBCYSNAAOMgQCgxVNZQKdj1Y5G/qlky7/HahBE
xYAAnjwyRVUtvbEczPrM7thlSNPywcA7
=nnCv
-----END PGP SIGNATURE-----

[Timmie (timmie)]

> I'm pretty sure SVN doesn't work over ftp, so I don't see how that is relevant.
Sorry about that. I was not to offend you.
It runs on filesystem and some specialized server set up.
And yes, up to now this FTP ability was my main pro for bzr!

Hope we get this fixed.
What can I do to provide more info?

[John A Meinel (jameinel)]

1) Why is the FTP service configured to allow a single connection per IP address? This at least seems overly restrictive.

2) In the code path in question, we have:
except ftplib.error_temp, e:
    if retries > _number_of_retries:
        raise errors.TransportError(msg="FTP temporary error during GET %s. Aborting."
                             % self.abspath(relpath),
                             orig_error=e)
    else:
        warning("FTP temporary error: %s. Retrying.", str(e))
        self._reconnect()
        return self.get(relpath, decode, retries+1)

There should be a message in .bzr.log which indicates what error we are getting.It might be that the server is giving us a "error_temp" error when really it should be giving a "permanent" error. (Like perhaps a file doesn't exist, but for some reason it is considering that a temporary error and not a permanent one.)

I don't think I've seen you declare what FTP server is being used.

I also haven't seen any comment about whether aftp:// has any effect (use active ftp rather than passive ftp.)

I'm not entirely sure why we reconnect for a temporary error. It might also be possible that we need to more actively *disconnect* as part of the 'reconnect()' code.

[Timmie (timmie)]

> Why is the FTP service configured to allow a single connection per IP address? This at least seems overly restrictive.
Sorry, I do not have any personal knowledge on FTP / http protocols nor on FTP server configurations. I am just using my account.
Please give me a list of information that I get forward to the IT department / server administrator.

I want to state that this has been working earlier (2-3 months ago) without problems.

> I don't think I've seen you declare what FTP server is being used.
I will ask.

> I also haven't seen any comment about whether aftp:// has any effect (use active ftp rather than passive ftp.)
I didn't change a thing.

I think that the server _IS_ very restrictive. I had this also with some FTP clients.
But when bzr is the first program to use FTP after booting then it should at least commit to the repository.

I normally use checkouts (no lightweight).

Thanks very much for looking into this! Highly appreciated.

[Martin Pool (mbp)]

2009/5/5 Tim Michelsen <email address hidden>:
> Hello,
> can you please help me here:
> ftp lib error: too many connections
> https://bugs.launchpad.net/bzr/+bug/365160

I wonder if Bazaar is actually trying to use two connections,
simultaneously, or if there is miscounting at some level.

It may be something like this: Bazaar discarded an ftp connection
object and started using a new one, but the old one was not
sufficiently closed for the server to recognize it as such. It might
be that we didn't send a QUIT message, or perhaps the socket was not
closed, or perhaps bzr in fact did do all of this and due to some
delay in the network the server had not actually seen it as closed
yet.

Tim, there is something you could do to help which would be to ask the
server administrator to send you the log lines corresponding to your
IP address, including the commands which were sent to the server.

Alternatively, and this may be slighly complicated, you can install
Wireshark on your machine and make a capture file of traffic to/from
that server. (It may contain your password so don't just post it here
or in the bug without marking the bug private.)

--
Martin <http://launchpad.net/~mbp/>

[Timmie (timmie)]

After attempting another commit to the remote repository, I issued netstat.

It shows that the ftp server address is under:
CLOSING/WAITING.

I do not know if this helps.

[Vincent Ladeuil (vila)]

Tim, it will really help if you followed the advices given in the comments, until we get more information (server log, wireshark trace, full .bzr.log (anonymized or bug marked private if the log mention passwords), etc), we can't do a good diagnosis.

If netstat shows a pending connection there are good chances that another process is till running (or can windows leave such sockets after the process death ?), are you sure you're not running TortoiseBzr or some bzr related GUI ?

[Timmie (timmie)]

I got into contact with the FTP server admin.

1) he asks not to use parallel connections
2) the server permist a maximum of 2 connections per IP (bzr apparently needs more!?)
3) he doe not see how parallel and "un-throttled" connections could increase speed

Hope that this helps.

[Robert Collins (lifeless)]

On Wed, 2009-07-08 at 23:01 +0000, Tim wrote:
> I got into contact with the FTP server admin.
>
> 1) he asks not to use parallel connections

FTP requires parallel connections. It uses 1 for control and 1 for data.

> 2) the server permist a maximum of 2 connections per IP (bzr apparently needs more!?)

bzr shouldn't need 3, but it does need the 2 that FTP requires. Its
possible we have a bug that causes more connections - we try to reuse
them but possibly we don't manage as well as we'd like.

> 3) he doe not see how parallel and "un-throttled" connections could increase speed

They don't, but we don't [intentionally] do that.

-Rob

[Martin Pool (mbp)]

2009/7/9 Tim <email address hidden>:
> I got into contact with the FTP server admin.
>
> 1) he asks not to use parallel connections
> 2) the server permist a maximum of 2 connections per IP (bzr apparently needs more!?)
> 3) he doe not see how parallel and "un-throttled" connections could increase speed
>
> Hope that this helps.

Hi Tim,

We're not intentionally using multiple connections for speed. There's
probably a bug causing one of them to be left open for too long.

If you get a network trace using eg wireshark it may indicate at what
point the error occurs. Something like this should do:

 sudo tshark -f 'host ftp.example.com'

As John points out above we're actually getting this during
_reconnect. I would guess this is where the problem occurs: bzr
thinks the connection has dropped and tries to start a new one, but
the server presumably thinks the old one was still open.

It looks like we actually failed twice: we couldn't connect, then we
tried again and that failure was treated as fatal.

Tim, as Vincent says, it's hard for us to make progress on this
unless you tell us what the server software is and/or provide a trace.

--
Martin <http://launchpad.net/~mbp/>

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Wed, 2009-07-08 at 23:01 +0000, Tim wrote:
>> I got into contact with the FTP server admin.
>>
>> 1) he asks not to use parallel connections
>
> FTP requires parallel connections. It uses 1 for control and 1 for data.
>
>> 2) the server permist a maximum of 2 connections per IP (bzr
> apparently needs more!?)
>
> bzr shouldn't need 3, but it does need the 2 that FTP requires. Its
> possible we have a bug that causes more connections - we try to reuse
> them but possibly we don't manage as well as we'd like.
>
>> 3) he doe not see how parallel and "un-throttled" connections could
> increase speed
>
> They don't, but we don't [intentionally] do that.
>
> -Rob
>

My guess is that we are getting a permissions failure or some other
random failure that is getting returned as an FTP "temporary" failure,
which makes us think that we should try reconnecting. And that
reconnection is failing.

1) Probably the server is returning a "temporary" failure for something
that is actually "permanent".
2) We probably aren't closing the previous connection before we try to
reconnect.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpWG1wACgkQJdeBCYSNAAOOdACcCd+YE+n8KwXDlM2Z1/cz/Jnu
LX0AnjzdDE1V6W0X8azScFSZCuqHc7HT
=5ukM
-----END PGP SIGNATURE-----

[Timmie (timmie)]

The error was:
error_temp: 421 Too many connections (2) from this IP

> sudo tshark -f 'host ftp.example.com'
The error occurs on the corporate network with Windows
I installed wireshark.
But I couldn't get any info out of it...

[Martin Pool (mbp)]

2009/7/10 Tim <email address hidden>:
> The error occurs on the corporate network with Windows
> I installed wireshark.
> But I couldn't get any info out of it...

What do you mean?

--
Martin <http://launchpad.net/~mbp/>

[Timmie (timmie)]

Hello again!
> I installed wireshark.
> But I couldn't get any info out of it...
I finally managed to detect trafffic with wireshark. The problem were the privileges.
See here: http://wiki.wireshark.org/CaptureSetup/CapturePrivileges This is not pointed to in the wireshark documentation.

What filter to you want be to apply or which protocol are you aiming at?
Can I flag this as a "security bug" if I do not want to disclose the logging info?

Thanks,
Timmie

[Martin Pool (mbp)]

2009/8/18 Tim <email address hidden>:
> Hello again!
>> I installed wireshark.
>> But I couldn't get any info out of it...
> I finally managed to detect trafffic with wireshark. The problem were the privileges.
> See here: http://wiki.wireshark.org/CaptureSetup/CapturePrivileges This is not pointed to in the wireshark documentation.
>
> What filter to you want be to apply or which protocol are you aiming at?

I want a capture of the ftp control data, which I think is port 21.

> Can I flag this as a "security bug" if I do not want to disclose the logging info?

You can make it private. However what I suggest you do is: get the
capture, click "follow stream" in wireshark, copy all the text from
there which should show the ftp commands, and then edit out anything
sensitive. Then attach that to this bug.

--
Martin <http://launchpad.net/~mbp/>

[Erin Vang (erin-vang)]

I am seeing this bug also, but I have found an explanation. I believe that the bug is actually one of failing to report the correct error.

- I am uploading projects to our ftp site, to a path like ftp.oursite.com/client/project
- my client is trying to do this:
    bzr checkout ftp://<email address hidden>:<email address hidden>/client/project
- we get the same errors as seen above, with traceback posted to new bug #440838 because I hadn't yet found this bug#
- the cause in my case is that my client's ftp account defaults to a different parent path, where he first lands in the "client" directory, which looks like / to him, whereas I land in that directory's parent. So the same folder has a different path for him than it does for me. The cure is for him to connect to the path that HE sees, viz.
    bzr checkout ftp://<email address hidden>:<email address hidden>/project

So, I do not think there's a bug, except that bzr is reporting the errors that it is when it SHOULD be reporting that the client doesn't have access to this ftp path.

[Martin Pool (mbp)]

Hi Erin - you're getting the same error "FTP temporary error: 421 Too
many connections (2) from this IP. Retrying. bzr: ERROR:
ftplib.error_temp: 421 Too many connections (2) from this IP" or
something very similar?

Thanks for looking into it.

--
Martin <http://launchpad.net/~mbp/>

[Martin Pool (mbp)]

We could in principle check that we only ever open one connection; that might save some time even for people who don't have this server restriction.