replicating to ubuntuone failed after successfully pairing with it

[Thomas Vander Stichele]

the pairing worked, but I don't think it replicated. I'm not sure how I can tell; is there a way I can see the couchdb stuff online In a raw way ?

Anyway, here's the log from desktopcouch attached.

[Thomas Vander Stichele (thomasvs)]

This ,by the way, is with httplib2 0.4.0

Will now try with 0.5.0

[Thomas Vander Stichele (thomasvs)]

With httplib2 0.5.0, I get this instead:

2009-11-24 17:17:43,148 DEBUG started replicating
2009-11-24 17:17:43,215 DEBUG replication of discovered hosts finished
2009-11-24 17:17:43,249 DEBUG static pairings are [('5d33b6ba-3026-4156-bb1f-5afa137d139e', 'ubuntuone', True, True)]
2009-11-24 17:17:43,249 DEBUG Looking up prefix for service 'ubuntuone'
2009-11-24 17:17:43,249 DEBUG Looking up prefix for service 'ubuntuone'
2009-11-24 17:17:43,251 ERROR replication of services aborted
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/desktopcouch/replication.py", line 161, in do_all_replication
    couchdb_io.mkuri("localhost", int(local_port))):
  File "/usr/lib/python2.6/site-packages/desktopcouch/pair/couchdb_pairing/couchdb_io.py", line 115, in get_database_names_replicatable
    all_dbs = set([db_name for db_name in couchdb_server])
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 127, in __iter__
    resp, data = self.resource.get('_all_dbs')
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 974, in get
    return self._request('GET', path, headers=headers, **params)
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 1010, in _request
    resp, data = _make_request()
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 1005, in _make_request
    body=body, headers=headers)
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 1101, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 887, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 859, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 948, in getresponse
    method=self._method)
  File "/usr/lib/python2.6/httplib.py", line 329, in __init__
    self.fp = sock.makefile('rb', 0)
AttributeError: 'NoneType' object has no attribute 'makefile'
2009-11-24 17:17:43,254 DEBUG finished replicating

So I still don't have it working. Any idea ?

[Thomas Vander Stichele (thomasvs)]

As Stuart asked me, went back to 0.4 which is what you guys deploy in ubuntu too.

After adding prints, it seems timeout is an 'object' (nothing more specific than that). Trying to figure out where it comes from.

[Thomas Vander Stichele (thomasvs)]

Alright, adding a __setattr__ to the HTTPSConnection object gives me this:

THOMAS: __set__ timeout <object object at 0xb7879490>
  File "/usr/lib/python2.6/threading.py", line 497, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/site-packages/twisted/python/threadpool.py", line 210, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.6/site-packages/desktopcouch/replication.py", line 148, in do_all_replication
    urlinfo = urlparse.urlsplit(str(remote_location))
  File "/usr/lib/python2.6/site-packages/desktopcouch/replication_services/ubuntuone.py", line 106, in __str__
    resp, content = client.request(url, "GET", headers=self.oauth_header)
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 975, in request
    conn = self.connections[conn_key] = connection_type(authority, timeout=self.timeout, proxy_info=self.proxy_info)
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 736, in __init__
    cert_file=cert_file, strict=strict)
  File "/usr/lib/python2.6/httplib.py", line 1065, in __init__
    HTTPConnection.__init__(self, host, port, strict, timeout)
  File "/usr/lib/python2.6/httplib.py", line 649, in __init__
    self.timeout = timeout
  File "/usr/lib/python2.6/site-packages/httplib2/__init__.py", line 723, in __setattr__
    import traceback; traceback.print_stack()

So, the culprit is my httplib.py, will now check there further.

[Thomas Vander Stichele (thomasvs)]

Well, this is unexciting:

    def __init__(self, host, port=None, strict=None,
                 timeout=socket._GLOBAL_DEFAULT_TIMEOUT):

even more unexciting, from socket.py

_GLOBAL_DEFAULT_TIMEOUT = object()

I hope there is a good reason for this peculiar code in the standard Python library, or why httplib pokes into a private var, and what the point is.

[Thomas Vander Stichele (thomasvs)]

ok well sigh now I figured out you guys carry a patch for httplib 0.4.0 that handles this. Fun.

Consider making it work with 0.5.0 as well, 0.4.0 predates python 2.6 hence your patching.

[Thomas Vander Stichele (thomasvs)]

ok, with your patch, getting slightly further.

However, now I get:

2009-11-28 17:58:33,291 DEBUG asking 'http://localhost:39949/' to replicate {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'w8sDs2jkXQzxgJ2DrLfp', 'consumer_key': 'ubuntuone', 'token_secret': '10RW9hrCCwRmC3T6TDlZq7lqLdZjShG4CRJ30BHqR4CLhmMxCfjCnftx86k1Nf2nkSqVFlxBp7fxLv01'}}} to contacts, using record {'source': {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'w8sDs2jkXQzxgJ2DrLfp', 'consumer_key': 'ubuntuone', 'token_secret': '10RW9hrCCwRmC3T6TDlZq7lqLdZjShG4CRJ30BHqR4CLhmMxCfjCnftx86k1Nf2nkSqVFlxBp7fxLv01'}}}, 'target': 'contacts'}

2009-11-28 18:07:05,704 ERROR can't replicate 'u/0a4/330/97870/contacts' 'http://localhost:39949/' <== {'source': {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'w8sDs2jkXQzxgJ2DrLfp', 'consumer_key': 'ubuntuone', 'token_secret': '10RW9hrCCwRmC3T6TDlZq7lqLdZjShG4CRJ30BHqR4CLhmMxCfjCnftx86k1Nf2nkSqVFlxBp7fxLv01'}}}, 'target': 'contacts'}
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/desktopcouch/pair/couchdb_pairing/couchdb_io.py", line 256, in replicate
    content=record)
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 981, in post
    **params)
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 1031, in _request
    raise ServerError((status_code, error))
ServerError: (500, ('case_clause', '{error,\n {{{badmatch,\n {error,\n {http_request_failed,\n <<"failed to replicate https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts/">>}}},\n [{couch_rep,do_init,1},\n {couch_rep,init,1},\n {gen_server,init_it,6},\n {proc_lib,init_p_do_apply,3}]},\n {child,undefined,"f5ed1d613a0c9ff4f94083b4c66dc43c",\n {gen_server,start_link,\n [couch_rep,\n ["f5ed1d613a0c9ff4f94083b4c66dc43c",\n {[{<<"source">>,\n {[{<<"url">>,\n <<"https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts">>},\n {<<"auth">>,\n {[{<<"oauth">>,\n {[{<<"consumer_secret">>,<<"hammertime">>},\n {<<"token">>,<<"w8sDs2jkXQzxgJ2DrLfp">>},\n {<<"consumer_key">>,<<"ubuntuone">>},\n {<<"token_secret">>,\n <<"10RW9hrCCwRmC3T6TDlZq7lqLdZjShG4CRJ30BHqR4CLhmMxCfjCnftx86k1Nf2nkSqVFlxBp7fxLv01">>}]}}]}}]}},\n {<<"target">>,<<"contacts">>}]},\n {user_ctx,<<"MPGTeQIAcm">>,[<<"_admin">>]}],\n []]},\n temporary,1,worker,\n [couch_rep]}}}'))
2009-11-28 18:07:05,706 DEBUG want to replipull 'gtd' from static host '1b67c630-f82e-453b-8b69-62cf94a0364e' @ couchdb.one.ubuntu.com
2009-11-28 18:07:05,732 DEBUG db exists, and we're ready to replicate
2009-11-28 18:07:05,739 DEBU...

[Thomas Vander Stichele (thomasvs)]

Reproduced basically the same problem on my 64-bit desktop, with all packages I built installed there and no local modifications left.

2009-11-29 12:52:58,229 DEBUG found 1 self_identity records
2009-11-29 12:52:58,282 INFO starting advertising _couchdb_location._tcp on port 39619
2009-11-29 12:52:58,293 DEBUG started replicating
2009-11-29 12:52:58,326 DEBUG replication of discovered hosts finished
2009-11-29 12:52:58,355 DEBUG static pairings are [('94b91e6d-526a-451b-90ec-09c3f296b4fe', 'ubuntuone', True, True)]
2009-11-29 12:52:58,355 DEBUG Looking up prefix for service 'ubuntuone'
2009-11-29 12:52:58,389 DEBUG Looking up prefix for service 'ubuntuone'
2009-11-29 12:52:58,416 INFO No o.u.c key. Maybe there's uo.c key?
2009-11-29 12:52:58,797 DEBUG skipping record {'push_to_server': True, 'ctime': '2009-11-29 12:30', 'service_name': 'ubuntuone', '_rev': '1-325606fe9016ba9435fc89a4165f6bd8', 'pairing_identifier': '94b91e6d-526a-451b-90ec-09c3f296b4fe', 'record_type': 'http://www.freedesktop.org/wiki/Specifications/desktopcouch/paired_server', '_id': 'caa2e294234d3f9362077ff3194d51e0', 'pull_from_server': True} with no excluded_names
2009-11-29 12:52:58,797 DEBUG found 0 excluded_names records
2009-11-29 12:52:59,196 DEBUG want to replipull 'contacts' from static host '94b91e6d-526a-451b-90ec-09c3f296b4fe' @ couchdb.one.ubuntu.com
2009-11-29 12:52:59,222 DEBUG db exists, and we're ready to replicate
2009-11-29 12:52:59,232 DEBUG asking 'http://localhost:39619/' to replicate {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'mpGfZfb645cHNr2XCHMK', 'consumer_key': 'ubuntuone', 'token_secret': 'K1FLr8MS8GctJxwzC7Q0ZFqkh9PzXP9J8Nb7rjRCnFTRvspFhSDcnlnzCVv2jsGrT76ZdGrxZZs48Lkf'}}} to contacts, using record {'source': {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'mpGfZfb645cHNr2XCHMK', 'consumer_key': 'ubuntuone', 'token_secret': 'K1FLr8MS8GctJxwzC7Q0ZFqkh9PzXP9J8Nb7rjRCnFTRvspFhSDcnlnzCVv2jsGrT76ZdGrxZZs48Lkf'}}}, 'target': 'contacts'}

2009-11-29 13:01:31,706 ERROR can't replicate 'u/0a4/330/97870/contacts' 'http://localhost:39619/' <== {'source': {'url': 'https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts', 'auth': {'oauth': {'consumer_secret': 'hammertime', 'token': 'mpGfZfb645cHNr2XCHMK', 'consumer_key': 'ubuntuone', 'token_secret': 'K1FLr8MS8GctJxwzC7Q0ZFqkh9PzXP9J8Nb7rjRCnFTRvspFhSDcnlnzCVv2jsGrT76ZdGrxZZs48Lkf'}}}, 'target': 'contacts'}
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/desktopcouch/pair/couchdb_pairing/couchdb_io.py", line 252, in replicate
    content=record)
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 981, in post
    **params)
  File "/usr/lib/python2.6/site-packages/couchdb/client.py", line 1031, in _request
    raise ServerError((status_code, error))
ServerError: (500, ('case_clause', '{error,\n {{{badmatch,\n {error,\n {http_request_failed,\n <<"failed to replicate https://couchdb.on...

[Thomas Vander Stichele (thomasvs)]

with my couchdb set to debug, I see this:

[Sun, 29 Nov 2009 17:01:32 GMT] [debug] [<0.164.0>] 'POST' /_replicate {1,1}
Headers: [{'Accept',"application/json"},
          {'Accept-Encoding',"identity"},
          {'Authorization',"OAuth realm=\"\", oauth_nonce=\"00906764\", oauth_timestamp=\"1259514092\", oauth_consumer_key=\"XlkBdOSjIX\", oauth_signature_method=\"HMAC-SHA1\", oauth_version=\"1.0\", oauth_token=\"mWSfpijFvs\", oauth_signature=\"UPZRzWg869DUa25m7Sk9CHoatsw%3D\""},
          {'Content-Length',"324"},
          {'Content-Type',"application/json"},
          {'Host',"localhost:48606"},
          {'User-Agent',"couchdb-python 0.6"}]

[Sun, 29 Nov 2009 17:01:32 GMT] [debug] [<0.164.0>] OAuth Params: [{"oauth_nonce","00906764"},
               {"oauth_timestamp","1259514092"},
               {"oauth_consumer_key","XlkBdOSjIX"},
               {"oauth_signature_method","HMAC-SHA1"},
               {"oauth_version","1.0"},
               {"oauth_token","mWSfpijFvs"},
               {"oauth_signature","UPZRzWg869DUa25m7Sk9CHoatsw="}]

[Sun, 29 Nov 2009 17:01:33 GMT] [debug] [<0.166.0>] didn't find a replication log for https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts/

[Sun, 29 Nov 2009 17:01:33 GMT] [debug] [<0.166.0>] didn't find a replication log for contacts

[Sun, 29 Nov 2009 17:01:33 GMT] [debug] [<0.170.0>] retrying couch_rep_httpc get request in 0.5 seconds due to {error, conn_failed}

[Sun, 29 Nov 2009 17:01:33 GMT] [debug] [<0.170.0>] retrying couch_rep_httpc get request in 1.0 seconds due to {error, worker_is_dead}

[Sun, 29 Nov 2009 17:01:34 GMT] [debug] [<0.170.0>] retrying couch_rep_httpc get request in 2.0 seconds due to {error, conn_failed}

[Sun, 29 Nov 2009 17:01:36 GMT] [debug] [<0.170.0>] retrying couch_rep_httpc get request in 4.0 seconds due to {error, worker_is_dead}

so something internally is failing, possibly the not finding the replication log.

How can I debug this further ?

[Thomas Vander Stichele (thomasvs)]

It did work on my macbook btw:

[Sun, 29 Nov 2009 17:27:32 GMT] [debug] [<0.1845.0>] found a replication log for https://couchdb.one.ubuntu.com/u%2F0a4%2F330%2F97870%2Fcontacts/

Where does it go look for these replication logs ?

[Matt Griffin (mattgriffin)]

Here's one place you can find out more about CouchDB logs.
https://wiki.ubuntu.com/UbuntuOne/Bugs#CouchDB

Sounds like this issue is resolved so I'm going to close it.