ftpfs crashes when connecting (4.7.0)

[SabreWolfy]

Binary package hint: mc

MC under Karmic worked fine. Upgrade to Lucid Beta with same INI file.

MC 4.7.0 crashes/hangs (difficult to tell) when connecting to FTP site, with error message about "ftpfs", "chdir first" and "strict rfc959". Could not connect regardless of whether "ftpfs_first_cd_then_ls=" was set to 0 or 1.

Connecting to FTP site in active or passive mode from command-line with "ftp" works, so the FTP site is working.

[Yury V. Zaytsev (zyv)]

What about posting the error message exactly as it appeared on the screen?

[SabreWolfy (sabrewolfy)]

grep -i ftp .mc/ini
ftpfs_directory_timeout=900
ftpfs_retry_seconds=30
ftpfs_always_use_proxy=0
ftpfs_use_passive_connections=0
ftpfs_use_unix_list_options=1
ftpfs_first_cd_then_ls=0
ftpfs_use_passive_connections_over_proxy=0
ftpfs_password=anonymous@
ftp_proxy_host=gate
ftp_codepage=Other_8_bit

Connect to ftp site in MC: Message scroll as it connects and then it stops with:

ftpfs: Reading FTP directory ftp... (strict rfc959)(chdir first)

I am behind a firewall and proxy, but I don't believe the proxy is required for FTP access. I can access the FTP site with or without the ftp_proxy variable set, in active or passive mode, with the "ftp" command on the command-line. MC pre-4.7.0 was also able to connect to the site with whatever default settings it had.

[Yury V. Zaytsev (zyv)]

Are you still able to reproduce it on my newer packages?

https://launchpad.net/~zyv/+archive/ppa

I remember we fixed an FTP breakage sometime around 4.7.0, but there was nothing about (strict rfc959)... It was an issue with the FTP servers requiring a password.

[SabreWolfy (sabrewolfy)]

Thanks. I've tested the latest release (ppa5), but the problem remains. I see messages scrolling about sending username and password, then it says (as before):

ftpfs: could not setup passive mode

and then hangs as before.

It still doesn't connect after I disable passive mode.

[Yury V. Zaytsev (zyv)]

You should NOT be able to connect to any FTP that is not supporting passive mode if you are behind the firewall (at least that does not do connection tracking). Does this happen with only this specific site (e.g. ftp.debian.org works, and it works for me!) or you are not able to connect to any site at all? Is this site public or not (e.g. you can disclose it so that I can try to reproduce it). Otherwise I'm afraid we will need logs...

[SabreWolfy (sabrewolfy)]

Behind a firewall proxy.

$ ftp_proxy=

$ echo $ftp_proxy

$ ftp ftp.debian.org
Connected to ftp.debian.org.
220 ftp.debian.org FTP server
Name (ftp.debian.org:mantis): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ^C
ftp> 221 Goodbye.

$ grep -i ftp .mc/ini
ftpfs_directory_timeout=900
ftpfs_retry_seconds=30
ftpfs_always_use_proxy=0
ftpfs_use_passive_connections=0
ftpfs_use_unix_list_options=1
ftpfs_first_cd_then_ls=0
ftpfs_use_passive_connections_over_proxy=0
ftpfs_password=anonymous@
ftp_proxy_host=
ftp_codepage=Other_8_bit

$ mc

Connect to ftp.debian.org. Hangs on "Reading ftp directory ..."

[SabreWolfy (sabrewolfy)]

I entered the details of the FTP site into Nautilus (Ctrl-L) as "ftp://<email address hidden>" and Nautilus prompted me for the password and logged into the FTP server. So I can connect via command-line 'ftp' or Nautilus, just not MC.

[Yury V. Zaytsev (zyv)]

Do you have a static external IP? I can privately provide you with an FTP server to get a log or alternatively if you can get a log yourself if you have an external box this would be of help.

[liontamer (geoff-lionhouse)]

I can confirm this bug exactly as SabreWolfy describes. It is still present in mc_4.7.0.5-1~lucid1~ppa3.

[Yury V. Zaytsev (zyv)]

So what? We can not reproduce it.

By the way, my mc configuration is different:

zyv@mypride:~/Desktop$ grep -i ftp ~/.mc/ini
ftpfs_directory_timeout=900
ftpfs_retry_seconds=30
ftpfs_always_use_proxy=0
ftpfs_use_passive_connections=1
ftpfs_use_passive_connections_over_proxy=0
ftpfs_use_unix_list_options=1
ftpfs_first_cd_then_ls=1
ftp_proxy_host=gate
ftpfs_password=anonymous@

Note

ftpfs_use_passive_connections=1
ftpfs_first_cd_then_ls=1

and the absence of

ftp_codepage=Other_8_bit

[liontamer (geoff-lionhouse)]

I hope my observation of this bug shows that it is real. The facts I can get are:
The ftp worked in Karmic and before (and still does) but not in Lucid.
I can ftp to the site using the program "ftp"
A message about password appears rapidly and is then replaced by "ftpfs: Reading FTP directory ... (chdir first)".
After a few seconds this is replaced by "ftpfs: Reading FTP directory ... (strict rfc959) (chdir first)"
The program then usually hangs with no response to mouse or keyboard.
Sometimes after a longer time the message "ftpfs: could not set up passive mode" appears and then the program hangs.
My ini file is the same as above except that I have "use_netrc=1" and "ftp_codepage=Other_8_bit". If I delete this last entry it is somehow automatically put back on exit from mc. How is this possible? I should add that I am not an expert.

[Silent Water (silentspam2000)]

I can confirm this bug on certain ftp servers. But not all - most work fine.

All ftp servers did work before the update and still are working from other ftp clients (with same connection parameters).

[Yury V. Zaytsev (zyv)]

You guys definitively get me wrong. I don't doubt that you are indeed encountering a problem, be it mc bug or something else. The fact is that I can't reproduce it, which means that nobody gonna fix it for you.

I asked you which FTP servers it does happen with right now and got no response. To diagnose a problem one would need to reproduce the problem and record the session with Wireshark to find out what's going on (or have control over such an FTP server in order to log the connection). There is no other way to fix this bug, OK?

[Serg (seansh)]

I want continue and confirm existing of this bug.
I use Ubuntu 10.04 and mc.
I need access to remote ftp over proxy in LAN.

mc 4.7.0.6-1 error on all sites (with/or without password)
mc 4.7.0 error (with/or without password)
mc 4.6.2 WORK OK !!!

My configuration is:
ftpfs_directory_timeout=900
ftpfs_retry_seconds=30
ftpfs_always_use_proxy=1
ftpfs_use_passive_connections=1
ftpfs_use_passive_connections_over_proxy=0
ftpfs_use_unix_list_options=1
ftpfs_first_cd_then_ls=1
ftp_proxy_host=10.0.0.95
ftpfs_password=anonymous@

"ftpfs_use_passive_connections_over_proxy" may be 0 or 1

List of messages:
ftpfs: sending...
ftpfs: reading directory ... (chdir first)
ftpfs: failed; nowhere to fallback to
mc displays red box "Cannot chdir to /#ftp:..." and stays in local directory
after 30s : ftps: sending... ftps: logged in
after 30s : ftps: disconnecting...

This situation you may check with Ubuntu 10.04 live dvd, mc 4.7.0, mc 4.6.2,
ftp site ex: archive.ubuntu.com,
access to Internet over Windows XP machine with proxy UserGate 2.8
(if you want, I can send you this version)

[Yury V. Zaytsev (zyv)]

I have no problems connecting and browsing archive.ubuntu.com . Did you read previous messages before posting? I need debug logs, that is FTP server log or wireshark conversation dump.

No logs ---> no fix.

[Serg (seansh)]

I am not big specialist in linux and I dont know how to make this log.
But I give you full information how to reproduce this bug.

If you want to see that, you need only:
1. PC # 1 + Ubuntu 10.04 live cd/dvd,
   (for clear situation, nothing installed)
2. mc-4.7.0.deb (from Ubuntu 10.04),
3. mc-4.6.2.deb (from Ubuntu 9.10),
4. PC # 2 with installed Windows XP
5. PC # 1 and PC # 2 connected over LAN
6. PC # 2 connected to Internet by 2nd Ethernet card
7. on PC # 2 installed proxy UserGate 2.8
   (I dont know, may be same situation with other proxy)
8. Try to access any FTP site from PC # 1
   (your joke about "archive.ubuntu.com" I dont understand)

MY PROBLEM IS SOLVED: I am returned to mc 4.6.2.
How about other people?

[Yury V. Zaytsev (zyv)]

The other people either don't have this problem (like me) or experience the problem and can't do anything about except for rolling back to 4.6.2.

I didn't make any jokes. I am telling you that I can browse archive.ubuntu.com with mc as the FTP client without any problem. I can not reproduce your problem if this is still unclear.

What you described below is not a way to reproduce the problem for the developer, it's your private setup. Where would I get extra PCs, Windows licenses, some obscure proxy server from???

Again, all of you guys, who have this problem: I NEED A WIRESHARK DUMP. Otherwise it's impossible to see what is going on. If you can not install wireshark on the machine with mc and record the packet dump during a failed attempt to establish an FTP connection for us to see what is going on I can't help you.

This might sound harsh, but sorry I have no time to do hand-holding.

[Serg (seansh)]

I send you wireshark's dump of 3 FTP sessions:
1 - successful for mc 4.6.2
and 2 unsuccessful for mc 4.7.0
(with ftpfs_use_passive_connections_over_proxy=0 and 1).
You may analyze it.

[Yury V. Zaytsev (zyv)]

Hey! This looks very cools. Thanks! I will have a look.

[Yury V. Zaytsev (zyv)]

It seems to be a bug in IPV6 support code.

[Yury V. Zaytsev (zyv)]

Could you please try my new package with additional patches (maverick or lucid or karmic) and tell whether it fixes the issue or not?

[Serg (seansh)]

I send you new dump for mc_4.7.0.9-2~lucid1~ppa2_i386.deb
There is 2 FTP sessions:
1 - unsuccessful (ftpfs_use_passive_connections_over_proxy=0)
1 - successful (...=1)
Note: mc 4.6.2 ignore this flag.

[Yury V. Zaytsev (zyv)]

Hmmm... What I can not understand is why 4.6.2 works for you.

It is true, that 4.6.2 ignores ftpfs_use_passive_connections_over_proxy, because this option was introduced later:

SHA1 ID: 2bba061bf0405acd4898d203b52e8f992c784fa5
Committer: Leonard den Ottolander <email address hidden> 2005-05-29 14:10:08

     * src/boxes.c, vfs/ftpfs.[ch]: Add checkbox to allow passive FTP
     over proxy to VFS dialog.

Also, for some strange reason, this option is off by default. BUT, before this patch, passive connections over proxy didn't happen at all:

     if (SUP.proxy)
- SUP.use_passive_connection = 0;
+ SUP.use_passive_connection = ftpfs_use_passive_connections_over_proxy;

This is confirmed by your Wireshark dump. On the other hand, with 4.7.* the very same active connection fails (?!), yet passive connection works.

I have to compare the dumps more carefully.

[Yury V. Zaytsev (zyv)]

Hmmm, few discoveries:

1) mc 4.7.x is not listening to the port it announces in active mode (bug which was not present in 4.6.2?!)

2) usergate does not seem to support extended passive mode correctly (answers RST, ACK instead on SYN, ACK)

3) usergate does not support EPRT, but it behaves correctly by executing subsequent commands.

=> it seems that my patch is not really needed. Maybe we can try PASV and PORT on IPV4 first though, and only if they fail try EPSV / EPRT just in case...

What is really needed is to find out why mc doesn't listen on local ports anymore.

[Yury V. Zaytsev (zyv)]

Do my new packages work for all your test cases? I hope I fixed this bug for good and the patch will make it in 4.7.0.10.

[SabreWolfy (sabrewolfy)]

Thanks! :)

GNU Midnight Commander 4.7.0.9 as downloaded from the PPA in comment #3 has solved the problem for me. Using Ubuntu 10.04.1.

[Yury V. Zaytsev (zyv)]

Well, then now you dudes say thanks to Serg, who finally provided me with the Wireshark dumps I asked for, which enabled to actually diagnose and fix the issue. The fate of the bug was in your hands :-)

I merged the patch upstream, so 4.7.5 and 4.7.0.10 won't have this problem anymore.

[Serg (seansh)]

New dump for mc_4.7.0.9-2~lucid1~ppa3_i386.deb
There is 2 successful FTP sessions:
(ftpfs_use_passive_connections_over_proxy=0 and =1)
Note: ~ppa2 works only when flag=1.

[Yury V. Zaytsev (zyv)]

Cool, looks like IPV4 is fixed for good. Would be nice to have someone with IPV6 to confirm that it's still working :-)

[liontamer (geoff-lionhouse)]

I can confirm that the fix works for me too. Thanks Yury and sorry that I was was expert enough to get the debug info you needed.

[liontamer (geoff-lionhouse)]

This bug has re-appeared on the upgrade from Natty to Oneiric. Symptoms the same as before. Sorry I am not expert enough to get wireshark output.

[Yury V. Zaytsev (zyv)]

Yes, unfortunately, this is to be expected. Oneiric carries the package from Debian, which doesn't include the patch that I produced. I have been planning to update the Debian package long ago, but unfortunately couldn't make enough time for that so far.

Let's hope I can finally get to it in the new year...

[liontamer (geoff-lionhouse)]

Thanks Yury for your rapid response. There are other ways of doing ftp but I love mc and use it for everything. I look forward to getting a new version.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mc - 3:4.8.1-2ubuntu1

---------------
mc (3:4.8.1-2ubuntu1) precise; urgency=low

  * Sync from debian testing (LP: #905610, LP: #314614, LP: #410031,
      LP: #562361, LP: #632816, LP: #713630, LP: #770673, LP: #837163)
  * Added Pre-Depends for dpkg-maintscript-helper availability

mc (3:4.8.1-2) unstable; urgency=low

  * Build-Depends are updated: 'bison' moved to build-deps;
    architecture wildcard replaced silly "type-handling | not+linux-gnu"
    (Closes: #587875 N:"Please remove type-handling dependency")
  * new patch to increase maximum file size for mcedit to 128 MiB
    (Closes: #369565 W:"mcedit: wishing a bigger file size limit")
    Thanks to Daniele Giacomini.
  * new patch to disable "Return does autoindent" by default in mcedit
    (Closes: #570502 N:"mcedit: adding extra spaces when pasting a text
     preceded by spaces")
    (Closes: #575711 N:"pasting extra tabs, AGAIN")
  * added mc.NEWS file with notes about important changes in this release
    (Closes: #661435 W:"lynx-like motion is lost during upgrade to 4.8.1")
  * added symlinks to all scripts in /usr/lib/mc for backward compatibility
  * new patch to correct path to scripts in man page
    (Closes: #661481 N:"Acknowledgement mc: /usr/share/mc/bin/mc.sh missing")
  * 'unzip' added to build-deps to set proper zip mode at build-time;
    'unzip' moved to Recommends from Suggests.
    (Closes: #661467 N:"mc: zip file browsing broken")
  * Recommends 'perl' and 'unzip' instead of Suggests
  * dropped old 20_wrong_path_to_wrappers.patch, which was breaking
    correct path to wrappers (note the precisely chosen file name ;)
  * corrected and properly annotated 09_uzip_broken_528239.patch

mc (3:4.8.1-1) unstable; urgency=low

  * New upstream release [December 2011]
    (Closes: #618542 N:"please follow upstream progress")
    (Closes: #528331 N:"[VFS] utar is unable to open .tar files")
    (Closes: #626287 N:"SHIFT+F6 should open rename dialog")
    (Closes: #609489 I:"If <F4> is pressed ~/.mc/cedit/Syntax is missing")
    (Closes: #606331 I:"regression: panel configuration on startup;
     view search configuration")
    (Closes: #567119 I:"mcedit ignores editnormal in MC_COLOR_TABLE")
    (Closes: #587372 N:"fish does not preserve modification time when
     copying files to remote host")
    (Closes: #592396 N:"file rename (F6) with non-usual characters failed")
    (Closes: #525146 N:"mc hangs when copying multiple files from ftp")
    (Closes: #574761 N:" [VFS] internal tar considers files containig
     '@' as directories.")
    (Closes: #584687 N:"mc/fish segfaults when remote copy/move appends
     to existing file")
    (Closes: #619092 W:"Wishlist: mc to open ISO files")
    (Closes: #602857 M:"use 7zr for generic .7z archives if available")
    (Closes: #61987 W:"total ETA wanted")
  * debian/watch
    • fixed and updated to fetch latest .tar.xz
  * dropped CDBS, now using debhelper only
  * debhelper & compat to version 9
  * dh-autoreconf to update toolchain
  * intltoolize to refresh Makefile.in.in
  * debian/control
    • standards to 3.9.3 (thanks to Andreas Tille)
    • added to build-deps:
      + 'type-handling...