ZEO client can't connect to Unix Domain socket

[Danny Navarro]

After upgrading from ZOBD3.9 to ZODB3.10 I couldn't connect to the ZEO server:

===============================
└─(%) paster serve mzcms.ini (mzcmsenv) [folders:365]─┘
Exception in thread Connect([(1, '/home/danny/sandbox/virtualenvs/mzcmsenv/mzcms-repo/folders/db/zeo.sock')]):
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/danny/sandbox/virtualenvs/mzcmsenv/lib/python2.6/site-packages/ZODB3-3.10.0-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py", line 392, in run
    success = self.try_connecting(attempt_timeout)
  File "/home/danny/sandbox/virtualenvs/mzcmsenv/lib/python2.6/site-packages/ZODB3-3.10.0-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py", line 416, in try_connecting
    wrappers = self._create_wrappers()
  File "/home/danny/sandbox/virtualenvs/mzcmsenv/lib/python2.6/site-packages/ZODB3-3.10.0-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py", line 457, in _create_wrappers
    for domain, addr in self._expand_addrlist():
  File "/home/danny/sandbox/virtualenvs/mzcmsenv/lib/python2.6/site-packages/ZODB3-3.10.0-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py", line 440, in _expand_addrlist
    for domain, (host, port) in self.addrlist:
ValueError: too many values to unpack
===============================

This is the mzcms.ini:
===============================
[DEFAULT]
debug = true

[app:zodb]
use = egg:mzcms#app
reload_templates = true
debug_templates = true
debug_authorization = false
debug_notfound = false
zodb_uri = zeo://%(here)s/db/zeo.sock?connection_cache_size=20000

[pipeline:main]
pipeline =
    egg:repoze.zodbconn#closer
    egg:Paste#evalerror
    egg:repoze.retry#retry
    egg:repoze.tm2#tm
    zodb

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 6543
===============================

My zeo.conf
===============================
%define INSTANCE .

<zeo>
  address $INSTANCE/db/zeo.sock
  read-only false
  invalidation-queue-size 100
  pid-filename $INSTANCE/db/zeo.pid
</zeo>

<blobstorage 1>
  <filestorage>
    path $INSTANCE/db/mzcms.db
  </filestorage>
  blob-dir $INSTANCE/db/blobs
</blobstorage>
===============================

[Danny Navarro (jdnavarro)]

I didn't include the tests in the patch because I don't know where to write it. If you give me a hint I can try.

[Tres Seaver (tseaver)]

I confirm that the bug exists: I believe it was introduced with the
patch adding IPV6 support.

The patch looks good to me. Any tests probably belongs in
'src/ZEO/tests/client-config.test'; unfortunately, the test jig there
is factored in such a way as to make reusing it for a non-TCP connection
difficult.

[Jim Fulton (jim-zope)]

On Tue, Oct 19, 2010 at 9:49 AM, Tres Seaver <email address hidden> wrote:
> I confirm that the bug exists:  I believe it was introduced with the
> patch adding IPV6 support.
>
> The patch looks good to me.  Any tests probably belongs in
> 'src/ZEO/tests/client-config.test';  unfortunately, the test jig there
> is factored in such a way as to make reusing it for a non-TCP connection
> difficult.

Also note that non-TCP tests need to be ommitted on windows.

Jim

--
Jim Fulton

[Danny Navarro (jdnavarro)]

For windows I guess I could handle the AttributeError exception that would be raised zrpc.client.ConnectionManager._guess_type when socket.AF_UNIX is not found. I could also raise the exception upfront earlier if windows is detected. I don't know what happens in windows when trying to use a unix domain address, I don't have a windows installation at hand.

I also saw that in order to test the unix sockets properly I would have to do some changes start_zeo_server in ZEO/tests/forker.py to support addresses of unix sockets.

Alternatively I could monkey patch the address of an already created TCP server ZEO server in ZEO/tests/client-config.test to look like a unix domain socket. But I don't know if that kind of test would be really useful.

Let me know what would you like to do about that and I will give it a try.

[Jim Fulton (jim-zope)]

On Wed, Oct 20, 2010 at 7:03 AM, Danny Navarro <email address hidden> wrote:
> For windows I guess I could handle the AttributeError exception that
> would be raised zrpc.client.ConnectionManager._guess_type when
> socket.AF_UNIX is not found. I could also raise the exception upfront
> earlier if windows is detected. I don't know what happens in windows
> when trying to use a unix domain address, I don't have a windows
> installation at hand.

I wasn't expecting you to make a windows error sane in this case.
That would be nice, but a bonus. :) If you do this, then
we'd need a windows test, which would be hard for you to
debug.

> I also saw that in order to test the unix sockets properly I would have
> to do some changes start_zeo_server in ZEO/tests/forker.py to support
> addresses of unix sockets.

If you use forker. I'm not sure that's necessary.
You could probably just instantiate a StorageServer
instance in a thread.

> Alternatively I could monkey patch the address of an already created TCP
> server ZEO server in ZEO/tests/client-config.test to look like a unix
> domain socket. But I don't know if that kind of test would be really
> useful.

I don't really follow this.

> Let me know what would you like to do about that and I will give it a
> try.

It would probably be easier (for both of us) for me to fix this. :)
If you want to get involved in ZEO development
and want a learning opportunity, I'm happy work with you.

Let me know.

Jim

--
Jim Fulton

[Danny Navarro (jdnavarro)]

On Wed, Oct 20, 2010 at 3:56 PM, Jim Fulton <email address hidden> wrote:
> On Wed, Oct 20, 2010 at 7:03 AM, Danny Navarro <email address hidden> wrote:
>> For windows I guess I could handle the AttributeError exception that
>> would be raised zrpc.client.ConnectionManager._guess_type when
>> socket.AF_UNIX is not found. I could also raise the exception upfront
>> earlier if windows is detected. I don't know what happens in windows
>> when trying to use a unix domain address, I don't have a windows
>> installation at hand.
>
> I wasn't expecting you to make a windows error sane in this case.
> That would be nice, but a bonus. :)  If you do this, then
> we'd need a windows test, which would be hard for you to
> debug.

For now I won't go for the bonus points. I've added to my TODO list to
setup a windows virtual machine and write an exception for UNIX domain
sockets in windows. I'll have more time next week.

>
>
>> I also saw that in order to test the unix sockets properly I would have
>> to do some changes start_zeo_server in ZEO/tests/forker.py to support
>> addresses of unix sockets.
>
> If you use forker.  I'm not sure that's necessary.
> You could probably just instantiate a StorageServer
> instance in a thread.
>
>> Alternatively I could monkey patch the address of an already created TCP
>> server ZEO server in ZEO/tests/client-config.test to look like a unix
>> domain socket. But I don't know if that kind of test would be really
>> useful.
>
> I don't really follow this.
>
>> Let me know what would you like to do about that and I will give it a
>> try.
>
> It would probably be easier (for both of us) for me to fix this. :)

OK, I'll start simple:

Index: src/ZEO/tests/client-config.test
===================================================================
--- src/ZEO/tests/client-config.test (revision 117779)
+++ src/ZEO/tests/client-config.test (working copy)
@@ -89,3 +89,21 @@
     104857600

     >>> storage.close()
+
+Make sure that a single string address, used for unix domain sockets, can
+also be used. Only applicable to non windows platforms.
+
+ >>> addr = '/tmp/zeosocket'
+ >>> storage = ZODB.config.storageFromString("""
+ ... <zeoclient>
+ ... server %s
+ ... wait false
+ ... </zeoclient>
+ ... """ % addr)
+
+ >>> import sys
+ >>> if not sys.platform.startswith('win'):
+ ... storage._addr
+ ['/tmp/zeosocket']
+
+ >>> storage.close()

> If you want to get involved in ZEO development
> and want a learning opportunity, I'm happy work with you.

I think it's a great opportunity. I could spend ~1 hour every day like
I'm doing these days but I don't know if my programming level will be
good enough for ZEO development with just 1 hour per day. I will
surely try.

Let me know if there is any other bug or something else you'd like me
to have a look at.

>
> Let me know.
>
> Jim
>
> --
> Jim Fulton
>
> --
> ZEO client can't connect to Unix Domain socket
> https://bugs.launchpad.net/bugs/663259
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Zope Object Database: Confirmed
>
> Bug description:
> After upgrading from ZOBD3.9 to ZODB3.10 I couldn't c...

[Danny Navarro (jdnavarro)]

Here I attach the patch with the test.

[Jim Fulton (jim-zope)]

For 3.10.1

[Lubomir Kostov (lkostov)]

This patch introduces Bug #1004513.

https://bugs.launchpad.net/zodb/+bug/1004513