Grokproject error for first time Windows users

[Steve Schmechel]

There is a rather hidden problem with Grokproject that occurs when
running it for the first time under the Windows OS without the win32api
module being available. It can also occur when win32api is available in
the system Python site-packages, but the user is working in a virtual
environment created using "virtualenv --no-site-packages".

This is related to the fix applied for the bug:
https://bugs.launchpad.net/grok/+bug/315223
Which addressed problems with Window paths with spaces.

The problem only occurs when initially creating the file "default.cfg"
in the default location of:
"C:\Documents and Settings\<USERNAME>\.buildout"

If the file does not exist and an explicit eggs_dir path is not
provided, an attempt is made to create the file.

  buildout_default = exist_buildout_default_file()
  if explicit_eggs_dir:
      # Put explicit_eggs_dir in the vars; used by the post command.
      vars['explicit_eggs_dir'] = explicit_eggs_dir
      vars['eggs_dir'] = (
          '# Warning: when you share this buildout.cfg with friends\n'
          '# please remove the eggs-directory line as it is hardcoded.\n'
          'eggs-directory = %s') % explicit_eggs_dir
  elif buildout_default:
      vars['eggs-dir'] = ''
  else:
      create_buildout_default_file()

However, the creation method does a conditional import of win32api which
fails.

  def create_buildout_default_file():
      default_dir = os.path.join(HOME, '.buildout')
      if not os.path.isdir(default_dir):
          os.mkdir(default_dir)
      eggs_dir = os.path.join(default_dir, 'eggs')
      if not os.path.isdir(eggs_dir):
          os.mkdir(eggs_dir)
      if sys.platform == 'win32':
          # Fix for paths with spaces on Windows.
          # See https://bugs.launchpad.net/grok/+bug/315223
          import win32api
          eggs_dir = win32api.GetShortPathName(eggs_dir)
      default_cfg = os.path.join(HOME, '.buildout', 'default.cfg')
      if not os.path.isfile(default_cfg):
          config_file = open(default_cfg, 'w')
          contents = """[buildout]
  eggs-directory = %s
  """ % (eggs_dir)
          config_file.write(contents)
          config_file.close()

After running grokproject and providing the user and passwd inputs, the
user receives:

  Enter user (Name of an initial administrator user): grok
  Enter passwd (Password for the initial administrator user):
  Downloading info about versions...
  Traceback (most recent call last):
    File "C:\vgrok\Scripts\grokproject-script.py", line 8, in <module>
      load_entry_point('grokproject==1.0.2', 'console_scripts', 'grokproject')()
    File "C:\vgrok\lib\site-packages\grokproject-1.0.2-py2.6.egg\grokproject\main.py", line 80, in main + extra_args)
    File "C:\vgrok\lib\site-packages\pastescript-1.7.3-py2.6.egg\paste\script\command.py", line 218, in run result = self.command()
    File "C:\vgrok\lib\site-packages\pastescript-1.7.3-py2.6.egg\paste\script\create_distro.py", line 125, in command vars = template.check_vars(vars, self)
    File "C:\vgrok\lib\site-packages\grokproject-1.0.2-py2.6.egg\grokproject\templates.py", line 117, in check_vars create_buildout_default_file()
    File "C:\vgrok\lib\site-packages\grokproject-1.0.2-py2.6.egg\grokproject\utils.py", line 45, in create_buildout_defau
  lt_file
      import win32api
  ImportError: No module named win32api

Would it be possible for Grokproject to check for the availability of
the module and either output a sensible warning to the user, or possibly
easy_install pywin32 to satisfy its own requirement?

The people that are likely to run into this will be first time users of
grokproject and buildout, so it will probably be a source of continuing
confusion and questions until the virtualenv developers start to handle
this upstream.

[Uli Fouquet (uli-gnufix)]

Thanks Steve for the report! I'll look into this tomorrow and hope to have a fix really soon as this could really be a nasty showstopper on win32.

If one of the more win32-oriented developers has a suggestion, please tell!

[Uli Fouquet (uli-gnufix)]

Had a quick view at the problem and considered several approaches.

Problematic code is::

    if sys.platform == 'win32':
          # Fix for paths with spaces on Windows.
          # See https://bugs.launchpad.net/grok/+bug/315223
          import win32api # <-- might fail
          eggs_dir = win32api.GetShortPathName(eggs_dir)

Possible solutions:

1) Get rid of win32 import and compute the path ourselves:

  Looks like this is not easily possible. The 8.3-path created by GetShortPathName() is apparently different on different
  win32 platforms. There is no publicly available algorithm, just a bunch of guesses that _might_ work on _some_
  platforms. No solid way thanks to proprietary code :-/ Maurits' solution (https://bugs.launchpad.net/grok/+bug/315223)
  seems to be the correct one.

2) Install win32 on-the-fly:

  Maybe this is possible but it would require very complicated extra-actions. Unfortunately the error happens at a place
  where we have no buildout environment available. Looks like a no-opt to me.

3) Provide win32 binaries with pywin32 included (distutils bdist_wininst).

  Might work, but requires a lot more effort on releases. Furthermore there seem to be problems on some platforms with
  distutils bdist_wininst. Not sure, whether resultung dists would work in virtual envs as well.

4) Just check on startup, whether we run on win32 and if so, whether win32 is installed. If not, abort with a sensitive
  error message, telling the user how to install pywin32.

For now I tend to do 4).

--
Uli

[Steve Schmechel (steveschmechel)]

Sound like that might be the only choice.

I had a closer look at the virtualenv bug that Michael Haubenwallner
sent me a link for:
https://bugs.launchpad.net/virtualenv/+bug/344512

It looks like they might come to the same solution,
"detect and inform".

--Steve

[Uli Fouquet (uli-gnufix)]

Hm, using setuptools, maybe one can really easy_install pywin32 like this (or similar)::

  from setuptools.command import easy_install

  # Fetch/install dist, register it by creating a .pth entry in site-packages/easy_install.pth
  easy_install.main(argv=['pywin32'])

  # That's not enough as the new .pth entry is not known to current system env.
  import site # does not reload 'site' really as it was already loaded on startup.
  reload(site) # Now site-packages/ should be scanned for new .pth entries
  import win32 # Should be importable now.

This would require pywin32 to be really easy-installable on win32.

Steve, can you confirm that this is the case?

If so, could some win user confirm that the above code really works in a fresh
(--no-site-packages) virtualenv?

--Uli

[Uli Fouquet (uli-gnufix)]

Finally, it looks like there _is_ a workaround for our problem that does not require pywin32:

If backslashes in the path written to default.cfg are doubled (only the backslashes, no other special chars like whitespaces and the like), buildout seems to cope with that path pretty well.

In other words:
   eggs-dir = C:\Documents and Settings\<USERNAME>\.buildout\eggs
   eggs-dir = "C:\Documents and Settings\<USERNAME>\.buildout\eggs"
   eggs-dir = 'C:\Documents and Settings\<USERNAME>\.buildout\eggs'
do not work, while
   eggs-dir = C:\DOCUME~1\<USERNAME>\BUILDO~1\eggs
   eggs-dir = C:\\Documents and Settings\\<USERNAME>\\.buildout\\eggs
do.

The latter path can of course be computed without win32api.

I changed the grokproject trunk accordingly and tested on a Win XP system, where it worked.

Steve, maybe you can confirm that it works on other (win32) platforms too?

To check you can follow the following steps:

1) create a virtual env with --no-site-packages

2) activate it

3) remove or rename
     C:\Documents and Settings\<USERNAME>\.buildout\default.cfg
    if it exists.

4) checkout trunk of grokproject

     Then, in the checkout:

4a) run python bootstrap\bootstrap.py

4b) run bin\buildout.exe

4c) run bin\grokproject sample
      It's essential to run grokproject from the local bin/ dir,
      not a grokproject maybe installed elsewhere.

If that installs a grokproject in sample/ dir, everything went well.

Any feedback would be highly appreciated.

--Uli

[Steve Schmechel (steveschmechel)]

I ran the latest trunk grokproject as you specified above, and it worked with your changes.
The directory and file were created in "Documents and Settings\<user>" correctly.
Grokproject successfully created the Sample project.

I had a few issues running the resulting Sample project, but I don't think it was due to your changes.
The problems were with particular zope packages and configuration and might have to do with
the pulled versions being incompatible with Python 2.6. I believe it was pulling vesion 1.0 of Grok
which I believe has issues with Python 2.6. I tried specifying 1.1a2 and it built but had different
problems at runtime.

I would be nice to try a new released Grokproject against a Grok 1.1a3 with latest changes from
JW and Souheil. I'm not sure that is planned though.

[Uli Fouquet (uli-gnufix)]

> I ran the latest trunk grokproject as you specified above, and it
> worked with your changes. The directory and file were created in
> "Documents and Settings\<user>" correctly. Grokproject successfully
> created the Sample project.

Great! That's the main point. I think we can close this bug now.
Thanks a lot Steve!

> I had a few issues running the resulting Sample project, but I don't
> think it was due to your changes. The problems were with particular
> zope packages and configuration and might have to do with the pulled
> versions being incompatible with Python 2.6. I believe it was pulling
> vesion 1.0 of Grok which I believe has issues with Python 2.6.

Exactly. 1.0 is not Python2.6-ready.

> I tried specifying 1.1a2 and it built but had different problems at
> runtime.

Hm, that _could_ be a problem (although this time not caused by
grokproject, I think). You could try specifying in buildout.cfg (of the
generated sample project):

extends =
   http://svn.zope.org/*checkout*/groktoolkit/trunk/grok.cfg

instead of 'versions.cfg'. That should fetch Grok 1.1a2 with an updated
list of dependent eggs/versions.

> I would be nice to try a new released Grokproject against a Grok 1.1a3
> with latest changes from JW and Souheil. I'm not sure that is planned
> though.

At least using an 'extends' like above should come near to what in the
end will be grok-1.1.cfg or grok-1.1a3.cfg, I think.

Before releasing grokproject I'd like to sort out two remaining things I
am not completely satisfied with. But that should be done until tomorrow
evening.

--Uli