git+ssh doesn't work on Windows

[speedwaystar]

Here's the log. Note that the push *seems* to have been successful, although it's hard to tell if the repository is complete.

$ bzr dpush git+ssh://<email address hidden>/speedwaystar/MobInfo2-FanUpdate.git
bzr: ERROR: select.error: (10038, 'An operation was attempted on something that is not a socket')

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 912, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1112, in run_bzr
  File "bzrlib\commands.pyo", line 690, in run_argv_aliases
  File "bzrlib\commands.pyo", line 705, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\foreign.pyo", line 329, in run
  File "bzrlib\branch.pyo", line 1068, in pull
  File "C:/Program Files/Bazaar/plugins\git\branch.py", line 525, in pull
  File "C:/Program Files/Bazaar/plugins\git\branch.py", line 474, in _update_revisions
  File "C:/Program Files/Bazaar/plugins\git\fetch.py", line 586, in fetch_objects
  File "C:/Program Files/Bazaar/plugins\git\remote.py", line 302, in fetch_objects
  File "C:/Program Files/Bazaar/plugins\git\remote.py", line 293, in fetch_pack
  File "C:/Program Files/Bazaar/plugins\git\remote.py", line 130, in fetch_pack
  File "C:\Program Files\Bazaar\plugins\git\_lib\dulwich\client.py", line 231, in fetch_pack
  File "C:\Program Files\Bazaar\plugins\git\_lib\dulwich\client.py", line 303, in can_read
  File "C:\Program Files\Bazaar\plugins\git\_lib\dulwich\client.py", line 268, in can_read
error: (10038, 'An operation was attempted on something that is not a socket')

bzr 2.2.1 on python 2.6.4 (Windows-post2008Server-6.1.7600)
arguments: ['C:\\Program Files\\Bazaar\\bzr.exe', 'dpush', 'git+ssh://<email address hidden>/speedwaystar/MobInfo2-FanUpdate.git']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.2.0]
  colo C:\Program Files\Bazaar\plugins\colo [0.1.0]
  dulwich C:\Program Files\Bazaar\plugins\dulwich [unknown]
  explorer C:\Program Files\Bazaar\plugins\explorer [1.1.1]
  fastimport C:\Program Files\Bazaar\plugins\fastimport [0.9.0dev]
  git C:\Program Files\Bazaar\plugins\git [0.5.3dev]
  keywords C:\Program Files\Bazaar\plugins\keywords [unknown]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.2.1]
  loom C:\Program Files\Bazaar\plugins\loom [2.2.1dev]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.2.1]
  news_merge C:\Program Files\Bazaar\plugins\news_merge [2.2.1]
  pipeline C:\Program Files\Bazaar\plugins\pipeline [unknown]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.19.2]
  rewrite C:\Program Files\Bazaar\plugins\rewrite [0.6.1]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.4]
  upload C:\Program Files\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Bazaar\plugins\xmloutput [0.8.6]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[Abscissa (gjmxbrzsxh)]

This is a fairly serious bug. It prevents windows users from being able to use their github repos (ie, writeable ones) with hg-git.

[speedwaystar (speedwaystar)]

indeed. as a result, i've moved to a different repository solution. i'm not sure that "triage" is the right response here.

[Abscissa (gjmxbrzsxh)]

This is a serious bug because it prevents windows users from being able to use their github repos with hg-git.

[Jelmer Vernooij (jelmer)]

The status is triaged, which means that the bug is confirmed and importance assigned in dulwich.

[Abscissa (gjmxbrzsxh)]

Sorry for my misunderstanding of "triaged". But considering that this is a blocker for any Windows user who uses gihub, gitorious or anything else that requires shh for writable repos, it's extremely difficult to imagine that "medium" would be an appropriate importance unless there happened to be a lot of data-corruption bugs (which could understandably take priority).

[Jose Raffucci (jose-raffucci)]

I would agree that upgrading the importance to high would be appropriate. I have found this to be a road block to using hg and git on windows. Please let us know how we can help squashing this bug quickly.

[Jelmer Vernooij (jelmer)]

This should now be fixed for bzr-git on Windows - please let me know whether this now works for you if you're using bzr-git on Windows.

[Jelmer Vernooij (jelmer)]

Hi Jose,

Patches to make dulwich support alternative SSH clients (such as Paramiko) are very welcome, or perhaps alternative implementations of can_read that work on Windows and don't rely on socket infrastructure.

[Gamlor (roman-stoffel)]

Hmm.

Tried it on Windows with Bazaar '2.3.1', bzr-git trunk, dulchwich trunk. Now it can connect to Github, but it hangs forever when checking out. It just uses the CPU but never progresses.

Command: bzr branch git+ssh://<email address hidden>:gamlerhart/db4opad.git

The output:
C:\temp\bazaar-stuff>bzr -D branch git+ssh://<email address hidden>:gamlerhart/db4opad.git
Connected (version 2.0, client OpenSSH_5.1p1)
Authentication (publickey) successful!
Secsh channel 1 opened.

Could it be that it only works with the latest Bazaar betas?

[Jose Raffucci (jose-raffucci)]

Hi Jelmer,

I use hg-git to work with git. I will take a look at the latest dulwich and try it out. I will look at helping u guys out with alternative clients or the can_read functions. I really appreciate your hard work. Will report back soon.

[Jelmer Vernooij (jelmer)]

Hi Gamlor,

Thanks for giving it a try. It should work with older versions of bzr as well, just as long as you have the current trunk revision of bzr-git.

[acelent (acelent)]

Hi Jelmer,

I developed a solution to the Win32 problem using a reader thread. I see that the solution you developed for bzr-git is much simpler. I didn't look at how can_read is supposed to work, but if it blocks in any way, it might result in what Gamlor reports. This is especially

I also fixed two more problems, which are the ability to run git.cmd and to avoid "abort: The handle is invalid" errors in thgw.exe by creating pipes for stderr as well, although we won't read from it. Although the latter works for local urls, it doesn't seem to work for ssh urls, even though I've removed a spurious "print" from hg-git in _ssh.py.

[acelent (acelent)]

Patch 1: Launch a reader thread in Win32 to implement can_read().

[acelent (acelent)]

Patch 2: Create stderr pipe to avoid invalid handle errors in thgw.exe.

[acelent (acelent)]

Patch 3: Allow running git.cmd.

[acelent (acelent)]

Ok, I just noticed that hg-git's _ssh.pt must also be updated accordingly. Not only the spurious print must be removed, but the subprocess wrapper class must also depend on the operating system and Popen must create a pipe for stderr.

I just tested it and it worked.

[Jelmer Vernooij (jelmer)]

It'd be great if windows users could provide some feedback on how well these patches work on Windows.

[Jelmer Vernooij (jelmer)]

I've pushed Paulo's alternative patch. Feedback on whether it works for git+ssh on Windows would be much appreciated.

[Gamlor (roman-stoffel)]

To me it looks like it doesn't work at all?

I still get the '[Error 2] The system cannot find the file specified' error. Used the bzr-git trunk and the dulwich trunk with bazaar 2.3.1. Is the patch on another branch?

[Martin Pool (mbp)]

I think it is on the bzr-git trunk. If you're still getting an error,
please post the traceback (which might be in .bzr.log, whose location
is shown in bzr --version).

[Jelmer Vernooij (jelmer)]

On 07/12/2011 01:41 PM, Gamlor wrote:
> To me it looks like it doesn't work at all?
>
> I still get the '[Error 2] The system cannot find the file specified'
> error. Used the bzr-git trunk and the dulwich trunk with bazaar 2.3.1.
> Is the patch on another branch?
>
The patch is in dulwich trunk; you'll need to install it manually to
test it.

Cheers,

Jelmer

[Gamlor (roman-stoffel)]

Ok. Yes I'm using dulwich and bzr-git trunk.

These are the commands I run:

Checkout:
bzr branch git://github.com/gamlerhart/db4opad.git db4opad

Push:
bzr dpush git+ssh://<email address hidden>:gamlerhart/db4opad.git

Error: ERROR: [Error 2] The system cannot find the file specified

The .bzr.log file:

Wed 2011-07-13 19:50:58 +0200
0.173 bazaar version: 2.3.1
0.174 bzr arguments: [u'dpush', u'git+ssh://<email address hidden>:gamlerhart/db4opad.git']
0.492 looking for plugins in C:/Users/Gamlor/AppData/Roaming/bazaar/2.0/plugins
0.492 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.970 encoding stdout as sys.stdout encoding 'cp850'
1.048 opening working tree 'C:/temp/bzrstuff/db4opad'
1.109 bzr-svn: using Subversion 1.6.6 (), subvertpy 0.7.5
1.261 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
1.262 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 926, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1126, in run_bzr
  File "bzrlib\commands.pyo", line 691, in run_argv_aliases
  File "bzrlib\commands.pyo", line 713, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\foreign.pyo", line 319, in run
  File "bzrlib\branch.pyo", line 1116, in lossy_push
  File "C:/Program Files (x86)/Bazaar/plugins\git\branch.py", line 893, in lossy_push
  File "C:/Program Files (x86)/Bazaar/plugins\git\branch.py", line 878, in push
  File "C:/Program Files (x86)/Bazaar/plugins\git\push.py", line 328, in dfetch_refs
  File "C:/Program Files (x86)/Bazaar/plugins\git\remote.py", line 352, in send_pack
  File "C:/Program Files (x86)/Bazaar/plugins\git\remote.py", line 156, in send_pack
  File "C:\Program Files (x86)\Bazaar\plugins\git\_lib\dulwich\client.py", line 158, in send_pack
  File "C:\Program Files (x86)\Bazaar\plugins\git\_lib\dulwich\client.py", line 384, in _connect
  File "C:\Program Files (x86)\Bazaar\plugins\git\_lib\dulwich\client.py", line 362, in connect_ssh
  File "subprocess.pyo", line 623, in __init__
  File "subprocess.pyo", line 833, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified

[Jelmer Vernooij (jelmer)]

Hi Gamlor,

You'll need to have an ssh client (such as plink) installed to be able to use git+ssh. dulwich will run it as "ssh".

[Gamlor (roman-stoffel)]

Ok it seems to work now. Thanks for all the help.

However looks like it has an issue with github? I get an 'ERROR: Repository not found.'. I will do a few experiments and tell report back.

[Jelmer Vernooij (jelmer)]

On Wed, 2011-07-20 at 09:55 +0000, Gamlor wrote:
> Ok it seems to work now. Thanks for all the help.
Great, thanks for confirming!

> However looks like it has an issue with github? I get an 'ERROR:
> Repository not found.'. I will do a few experiments and tell report
> back.
What is the URL you are using ?

Cheers,

Jelmer

[Gamlor (roman-stoffel)]

This URL: git+ssh://<email address hidden>:gamlerhart/db4opad.git

[Jelmer Vernooij (jelmer)]

On Wed, 2011-07-20 at 10:30 +0000, Gamlor wrote:
> This URL: git+ssh://<email address hidden>:gamlerhart/db4opad.git
You probably want:

git+ssh://<email address hidden>/gamlerhart/db4opad.git

Cheers,

Jelmer

[Jelmer Vernooij (jelmer)]

Marking the bzr-git part of this bug invalid, as it didn't require any changes outside of dulwich.

[Gamlor (roman-stoffel)]

Just want to confirm that is works now. Thanks for all the help and fixes =)