desktopcouch wont replicate "dmedia" DB

[Jason Gerard DeRose]

Binary package hint: desktopcouch

Say it ain't so, desktopcouch!

I'm trying to replicate the "dmedia" database containing metadata of all we shot in Budapest, replicate from my laptop to U1 to my desktop so I can do the first device-to-device file transfers (exciting!)

But something is borked with syncing the "dmedia" database in particular. This is what's happening on my desktop (which currently contains an empty "dmedia" DB):

"""
2011-05-22 02:50:30,579 ERROR can't replicate 'u/69b/a53/210959/dmedia' 'http://localhost:51443/' <= {'url': 'https://couchdb.one.ubuntu.com/u%2F69b%2Fa53%2F210959%2Fdmedia', 'auth': {'oauth': {u'consumer_secret': 'HiddenHidd', u'token': u'ZWR7LVTnBb81Fj6rcj4b', u'consumer_key': u'ubuntuone', u'name': u'UbuntuOne token for https://ubuntuone.com', u'token_secret': 'HiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHiddenHi'}}} to 'dmedia'
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/desktopcouch/application/pair/couchdb_pairing/couchdb_io.py", line 319, in replicate
    data = local_server.replicate(source, target)
  File "/usr/lib/pymodules/python2.7/couchdb/client.py", line 214, in replicate
    status, headers, data = self.resource.post_json('_replicate', data)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 399, in post_json
    status, headers, data = self.post(*a, **k)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 381, in post
    **params)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 419, in _request
    credentials=self.credentials)
  File "/usr/lib/pymodules/python2.7/desktopcouch/records/http.py", line 259, in request
    raise ServerError((status, error))
ServerError: (500, ('json_encode', '{bad_term,<0.1716.0>}'))
2011-05-22 02:50:30,581 DEBUG finished replicating
"""

The full desktop-couch-replication.log is attached.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: desktopcouch-ubuntuone 1.0.7-0ubuntu2
ProcVersionSignature: Ubuntu 2.6.38-8.42-generic 2.6.38.2
Uname: Linux 2.6.38-8-generic x86_64
Architecture: amd64
CheckboxSubmission: fdbdfcded0c0bb479a6b52e9ec5af131
CheckboxSystem: edda5d4f616ca792bf437989cb597002
Date: Sun May 22 02:42:19 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007.1)
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US:en
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: desktopcouch
UpgradeStatus: Upgraded to natty on 2011-01-30 (111 days ago)

[Jason Gerard DeRose (jderose)]

Attached dmedia.couch as it currently exists on my laptop... not sure if this is ever getting synced correctly to U1 in the first place.

[Jason Gerard DeRose (jderose)]

This is replication log from my laptop, can perhaps shed light on what might be happening at other end.

[Roman Yepishev (rye)]

Hello,

It looks like occasionally dmedia database is synced (that's the output of ubuntuone-desktopcouch-diag.py from ubuntuone-scripts):

--- dmedia ---
 dmedia -> https://couchdb.one.ubuntu.com/u%2F69b%2Fa53%2F210959%2Fdmedia:
 8 attempts, 3 failed (3 seen), 5 succeeded (62%)
 Successfully synchronized @ 2011-05-22 10:53:18,431

 https://couchdb.one.ubuntu.com/u%2F69b%2Fa53%2F210959%2Fdmedia -> dmedia:
 5 attempts, 2 failed (0 seen), 3 succeeded (60%)
 Successfully synchronized @ 2011-05-22 10:53:58,411

Could you please check whether the update_seq of dmedia documents in couchdb.one.ubuntu.com is the same as your local system:
Download http://people.canonical.com/~roman.yepishev/us/ubuntuone-couchdb-query and run it as
python ubuntuone-couchdb-query dmedia

Open futon via ~/.local/share/desktop-couch/couchdb.html and use direct access to dmedia database - http://localhost:someport/dmedia - the same output as with ubuntuone-couchdb-query will appear.

[Eric Casteleijn (thisfred)]

If it does keep consistently failing, you may want to look at recent document edits/additions, to see whether you've hit this bug in couchdb itself:

https://issues.apache.org/jira/browse/COUCHDB-1176

[Jason Gerard DeRose (jderose)]

Thanks Roman, thanks Eric.

To test again, I deleted the "dmedia" DB on my workstation. desktopcouch recreated it the next time it tried to sync "dmedia" from U1 (so I have an empty "dmedia" with update seq 0), and then I get this error in the log:

  ResourceNotFound: ('db_not_found', 'could not open https://couchdb.one.ubuntu.com/u%2F69b%2Fa53%2F210959%2Fdmedia/')

I was trying to think of what's different about the "dmedia" DB compared to other DB that are syncing okay:

1) dmedia is bigger (about 15 MB)
2) dmedia has more docs (3441)

Not sure if that could be causing problems or not.

Please let me know what other info you need, other things I should test.

Thanks!

[Jason Gerard DeRose (jderose)]

Oh, forgot to run ubuntuone-couchdb-query. Here's output:

jderose@jgd-ws:/tmp$ python ubuntuone-couchdb-query dmedia
{"db_name":"u/69b/a53/210959/dmedia","doc_count":3441,"doc_del_count":0,"update_seq":3443,"purge_seq":0,"compact_running":false,"disk_size":12857443,"instance_start_time":"1306169436917235","disk_format_version":5,"committed_update_seq":3443}

Current update seq on my laptop is 8787, so looks like laptop hasn't successfully synced in a bit. But shouldn't my workstation still be able to replicated down the most recent "dmedia" in U1?

Should I delete dmedia in U1, try to start again?

[Eric Casteleijn (thisfred)]

All your machines should replicate completely independently from eachother, yes. So if neither of your machines are replicating successfully, while other databases are, something about the dmedia database is breaking replication. You could try removing it locally, to see if that has an effect, but if the above is true, it might not.

We may have to ask the couch developers to see if they can make any sense of the couchdb logs, in that case.

[Jason Gerard DeRose (jderose)]

One thing to note is that I can replicate from desktopcouch to the system-wide CouchDB (I do this often for testing). So it doesn't seem like the problem is CouchDB being fundamentally unable to replicate this DB, but more an issue with network timeouts or something when replicating to the remote U1 infrastructure.

Any ideas about other tests I could run?

Is there an easy way for you U1 folks to delete all the databases I currently have in U1 so I can test this with a fresh start?

[Eric Casteleijn (thisfred)]

There are load problems on at least one of the couchdb servers, which we are investigating with the assistance of couch one, so that could be one possible cause, though I would expect less consistent failure in that case. Note that what we have running on our servers is not 100% the same as what is in Ubuntu, since we've had to patch some of the authentication code for our specific set up, so it could also be that you're hitting a bug in those patches.

If you have ubuntuone-couch installed, you can delete your own databases in the cloud, which is as simple as:

u1couchquery --http-method=DELETE [DBNAME]

u1couchquery can be used to test access to all your couch data in the cloud, and ubuntuone-couch also contains the lower level command u1oauthrequest that you can use to do oauth signed requests to any url (it will use your u1 credentials by default, so it takes some of the annoying details off your hands.)

[Jason Gerard DeRose (jderose)]

Thanks for the tip, Eric!

One problem I'm having is that I can't seem to delete the "gwibber_accounts" and "gwibber_preferences" databases. I know that these are being blocked from replication, but they still get created on all my machines, and desktopcouch still tries to replicate to/from these DB.

Is there anyway for someone to manually delete these? Or have the U1 infrastructure delete them for everyone? Would probably reduce your load a bit too, so bonus there.

[Jason Gerard DeRose (jderose)]

Okay, another little tidbit I've observed: it looks like the replication of "dmedia" is failing upon trying to replicate the first doc with attachments... failing with a JSON encoding error:

"""
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/desktopcouch/application/pair/couchdb_pairing/couchdb_io.py", line 319, in replicate
    data = local_server.replicate(source, target)
  File "/usr/lib/pymodules/python2.7/couchdb/client.py", line 214, in replicate
    status, headers, data = self.resource.post_json('_replicate', data)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 399, in post_json
    status, headers, data = self.post(*a, **k)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 381, in post
    **params)
  File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 419, in _request
    credentials=self.credentials)
  File "/usr/lib/pymodules/python2.7/desktopcouch/records/http.py", line 259, in request
    raise ServerError((status, error))
ServerError: (500, ('json_encode', '{bad_term,<0.19787.0>}'))
"""

Does that sound familiar?

[Eric Casteleijn (thisfred)]

We've been seeing it quite a lot recently, but I have not figured out what it means yet. It *looks* like replication results in an error on either the server or the client which does not return a nice (json encodable) error message, but instead an erlang process id.

Could you maybe attach the couchdb logs, or the part of them that pertains to this error if you can find it, (keeping in mind that couchdb uses UTC for its logging time.) Take care to mark the bug as private first if there is anything sensitive in there like oauth secrets, or scrub those out first.

The couchdb logs live in ~/.cache/desktopcouch/

[Eric Casteleijn (thisfred)]

Marking this as a duplicate, please add your logs at https://bugs.launchpad.net/ubuntuone-servers/+bug/665024