Ubuntu

[FFE] Please merge couchdb 1.1.0-1 (main) from Debian unstable (main)

Reported by Mat Scales on 2011-07-28
126
This bug affects 28 people
Affects Status Importance Assigned to Milestone
couchdb (Ubuntu)
High
Ubuntu One Client Engineering team
Oneiric
High
Mat Scales
Precise
High
Ubuntu One Client Engineering team

Bug Description

There is a new version of couchdb in debian sid that should be merged into Ubuntu

This needs a freeze exception for Oneiric, as we are past feature freeze and this is a major version update. However, it fixes significant regressions caused by the newer version of erlang being in Oneiric. I would have marked bug #780972 as a duplicate of this one, however it has too many duplicates already, and Launchpad gives a timeout error when attempting to do so.

I will also file another bug separately, requesting demotion of couchdb and its dependents, to universe, nullifying any MIR necessities for this to be fixed.

Related branches

lp:~wibblymat/ubuntu/oneiric/couchdb/update-to-1.1.0
James Page: Needs Fixing on 2011-10-25
Daniel Holbach: Pending requested 2011-09-06
Ubuntu branches: Pending requested 2011-09-06
Mat Scales (wibblymat) on 2011-07-28
Changed in couchdb (Ubuntu):
assignee: nobody → Mat Scales (wibblymat)
status: New → In Progress
Changed in couchdb (Ubuntu):
importance: Undecided → Wishlist
John Lenton (chipaca) wrote :

As per lp:837190, this (1.1.0, or fixes from 1.0.3) is needed for couchdb to work with the version of erlang in oneiric.

Changed in couchdb (Ubuntu):
importance: Wishlist → High
Jason Gerard DeRose (jderose) wrote :

So following chipaca's advice, I've built and installed 1.1.0. I've been beating up on it with the dmedia unit tests. I haven't encountered any failures, but there seems to be a dramatic performance regression in 1.1.0.

For example, from 1.0.1 (plus Ubuntu patches) to this 1.1.0 package:

Microfiber benchmark: 198 => 24.5 ops/second

PyCouchDB benchmark: 55.3 => 18.4 ops/second

== Running against CouchDB 1.0.1 ==

jderose@jgd-test:~/bzr/microfiber/trunk$ python3 benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 12.10
    Saves per second: 165.2
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 6.21
    Gets per second: 321.9
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 11.99
    Deletes per second: 166.8
Total seconds: 30.30
Total ops per second: 198.0

jderose@jgd-test:~/bzr/microfiber/trunk$ python benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 83.59
    Saves per second: 23.9
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 10.99
    Gets per second: 182.1
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 13.92
    Deletes per second: 143.7
Total seconds: 108.50
Total ops per second: 55.3

== Running against CouchDB 1.1.0 ==

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 82.96
    Saves per second: 24.1
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 80.03
    Gets per second: 25.0
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 82.34
    Deletes per second: 24.3
Total seconds: 245.33
Total ops per second: 24.5

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 161.63
    Saves per second: 12.4
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 80.52
    Gets per second: 24.8
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 83.73
    Deletes per second: 23.9
Total seconds: 325.88
Total ops per second: 18.4

Jason Gerard DeRose (jderose) wrote :

Okay, if you uncomment line 26 in /etc/couchdb/default.ini:

http://bazaar.launchpad.net/~wibblymat/ubuntu/oneiric/couchdb/update-to-1.1.0/view/head:/etc/couchdb/default.ini.tpl.in#L26

socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

Then the performance is back up to what is was in 1.0.1. I guess CouchDB was perhaps running with nodelay by default before, and now it isn't? Anyway, gory benchmark details running with that line uncommented:

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 13.62
    Saves per second: 146.9
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 6.31
    Gets per second: 316.9
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 10.27
    Deletes per second: 194.8
Total seconds: 30.20
Total ops per second: 198.7

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 84.01
    Saves per second: 23.8
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 9.09
    Gets per second: 220.0
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 11.65
    Deletes per second: 171.7
Total seconds: 104.75
Total ops per second: 57.3

Jason Gerard DeRose (jderose) wrote :

To help this get more testing, I went ahead and built it in the Novacut Daily Builds PPA:

https://launchpad.net/~novacut/+archive/daily?field.series_filter=oneiric

https://twitter.com/#!/novacut/status/108805962009296896

I've been beating up on it pretty hard, haven't encountered any problems yet. After further thought, perhaps we don't want to change /etc/couchdb/default.ini, should instead keep it the way upstream ships. But regardless, we should make sure desktopcouch is launching CouchDB with:

[httpd]
socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

In one of its config files. Perhaps we should add it to the other static file desktopcouch includes:

  /etc/xdg/desktop-couch/compulsory-auth.ini

Jason Gerard DeRose (jderose) wrote :

Okay, think I'm encountering a problem. After running for a while, CouchDB seems to hang when a request is made to a specific database. My experimental dc3 has a monitor that once a minute requests the welcome string like this:

  GET /

Requesting the welcome string continued to work fine. However requests like this would hang:

  GET /dmedia

I restarted dc3 and then everything worked fine again. I think I encountered the same thing testing against desktopcouch, but I didn't narrow things down as nicely. I'll see if I can reliably reproduce this, although it seems to only happen after CouchDB has been running several hours.

John S. Gruber (jsjgruber) wrote :

I tried the couchdb-bin from the Novacut ppa. It seemed to work fine and solved LP: #780972. After trying it and then going back to the current Oneiric version I found that I soon couldn't access my database (lernid preferences). I had to delete the database and recreate it to go on==not a big deal for Lernid, but something to be aware of in terms of migration planning if it is the general situation, particularly for contacts and bookmarks.

Rodney Dawes (dobey) on 2011-09-08
summary: - Please merge couchdb 1.1.0-1 (main) from Debian unstable (main)
+ [FFE] Please merge couchdb 1.1.0-1 (main) from Debian unstable (main)
Chris Coulson (chrisccoulson) wrote :

Tracking in oneiric, as this currently blocks bug 743096

Rodney Dawes (dobey) on 2011-09-08
description: updated
Rodney Dawes (dobey) wrote :

I have filed bug #844995 requesting the demotion of couchdb and its dependents to universe from main.

John S. Gruber (jsjgruber) wrote :

Since it's late in the cycle, I've proposed to merge a cherry picked fix for LP: #780972.

Martin Pitt (pitti) wrote :

As ubuntuone is pretty much the only (important) rdepends of couchdb in oneiric now, it would not be an unnecessarily big risk to update it now. My main concerns here are the performance regression (which seems to be just a question of where to apply the workaround/fix you mentioned, in couchdb or in desktopcouch), and the mentioned crash. If such a crash didn't happen with the current version, that does seem to be a reason for investigating and fixing it first before upgrading?

Is there anything else except for the cherrypicked fix for #780972 which you need for oneiric? If so, I'm fine with going 1.1 if the crash gets fixed, otherwise a cherrypick sounds appropriate at this time.

Jason Gerard DeRose (jderose) wrote :

Martin,

I haven't again encountered the crash I mentioned. It occurred using my experimental desktopcouch-like `dc3`, so there's a good chance I was doing something stupid :P

I feel I've tested CouchDB 1.1.0 pretty comprehensively, aside from one key area: apps that use desktopcouch through libdesktopcouch-glib-1.0-2, libcouchdb-glib-1.0-2 (evolution-couchdb is probably the most widely used consumer). If we could get some testing via evolution-couchdb, and make sure U1 sync works okay still using it, then I personally feel pushing CouchDB 1.1.0 into Oneiric is a reasonable move.

As far as cherry picking, the CouchDB 1.0.3 release notes mention three Erlang R14B02 fixes:

* Fixed compatibility issues with Erlang R14B02
* Fixed error when restarting replications in OTP R14B02
* Fixed OAuth signature computation in OTP R14B02

BTW, the Novacut components all talk to CouchDB directly, so they're kinda weird cases in desktopcouch land... although still good testing.

Martin Pitt (pitti) wrote :

Thanks Jason for the followup. So I'm fine with a full upgrade if the U1 team will sort out the MIR and performance regression, and I'm also fine with cherrypicking the three patches.

Rodrigo Moya (rodrigo-moya) wrote :

I have been, for the whole Oneiric cycle, testing evo-couchdb with the system-wide CouchDB instance (that is, no auth), and everything worked ok. So I'd be happy to test dc with the new couchdb and fix any issue that comes up. Although I think it should just work if the "unauthenticated" crashes are fixed. But anyway, just let me know where to get an updated couchdb package from, and I can do a round of testing before/after you upload to the archive

On Fri, 09 Sep 2011 08:49:01 -0000, Rodrigo Moya <email address hidden> wrote:
> I have been, for the whole Oneiric cycle, testing evo-couchdb with the
> system-wide CouchDB instance (that is, no auth), and everything worked
> ok. So I'd be happy to test dc with the new couchdb and fix any issue
> that comes up. Although I think it should just work if the
> "unauthenticated" crashes are fixed. But anyway, just let me know where
> to get an updated couchdb package from, and I can do a round of testing
> before/after you upload to the archive

you can build the package off of the merge proposal associated with this
bug. Or, https://launchpad.net/~novacut/+archive/daily :)

primefalcon (primefalcon) wrote :

I hope this gets fixed quick

primefalcon (primefalcon) wrote :

I have having this issue due to couchdb accessing google contacts everytime I open thunderbird

Rodney Dawes (dobey) wrote :

I don't know what issue you are having, as you haven't actually described one. But this bug is a sync request for a new version of couchdb, and couchdb doesn't access google contacts itself. If you have an issue, please open a new bug.

tags: added: rls-mgr-o-tracking
primefalcon (primefalcon) wrote :

well the issue I am having is thunderbird constantly throwing this error...

There was a problem opening the address book "my <email address hidden>" - the message returned was: Cannot open book: Cannot process, book backend is opening.

I subscribed to this bug: https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280, which was closed with this message: Marking as a duplicate of bug 817656, as that should fix the current desktopcouch breakage which causes this error

Rodney Dawes (dobey) wrote :

We don't need 1.1.0 for Oneiric, as the cherrypicked patch in couchdb 1.0.1-0ubuntu17 has fixed the desktopcouch issue. But we should get the latest version from debian synced over as soon as the P archive opens up after 11.10 release.

Martin Pitt (pitti) wrote :

Thanks Rodney, setting this for P then.

Changed in couchdb (Ubuntu Oneiric):
status: In Progress → Won't Fix
Kate Stewart (kate.stewart) wrote :

I've removed the duplicate from https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280. Since this is still a problem in Oneiric based on the above comments have occured after couchdb 1.0.1-0ubuntu17 landed on Sept. 13th, and issues were seen after that (see comment #18)

tags: added: rls-mgr-p-tracking
removed: rls-mgr-o-tracking
primefalcon (primefalcon) wrote :

its still happening with a slightly different message, it says backend is opening now rather than missing, I included further info here: https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280/comments/14

Rodney Dawes (dobey) wrote :

Brad (primefalcon), your error seems to have nothing to do with couchdb, but rather is the same issue occuring within the gmail backend.

Jason Gerard DeRose (jderose) wrote :

So how soon can this be merged into Precise? Is there anything I can do to help (as a person without upload rights)?

Stefano Rivera (stefanor) wrote :

Someone just needs to do it. No upload rights required (wibblymat doesn't have upload rights either). Just prepare the merge and get it into the sponsorship queue https://wiki.ubuntu.com/SponsorshipProcess

Stefano Rivera (stefanor) wrote :

Unsubscribing the release team. This FFe didn't happen.

Daniel Holbach (dholbach) wrote :

https://code.launchpad.net/~wibblymat/ubuntu/oneiric/couchdb/update-to-1.1.0/+merge/74236 merges 1.1.0-1 from Debian, but as 1.1.0-2 is in testing now (and seems to resolve a number of regressions), it'd be great if somebody went ahead and updated the merge to 1.1.0-2.

Jason Gerard DeRose (jderose) wrote :

Perhaps this bug should be closed? As there probably wont be as many Ubuntu specific changes going forward, I think a sync from Debian is more appropriate than a merge. Also up for debate is whether CouchDB should move from main to universe. Either way, the sooner this could happen the better. Also, Novacut says "pretty please" :P

I filled a sync-request bug here - https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/817656

And I personally think this bug here should be closed as Wont Fix.

Jason Gerard DeRose (jderose) wrote :

Oops, never mind... didn't realize Debian didn't have the couchdb/couchdb-bin split.

Merge it is, but should be 1.1.1-1 from Debian testing.

Jason Gerard DeRose (jderose) wrote :

Er, had the bug wrong too, my aim is way off tonight - https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/902417

That's the sync request. And to clearly correct the record, please don't sync from Debian... Novacut really needs the couchdb-bin package as we don't want to force our users to be running a system-wide CouchDB for no reason.

Jason Gerard DeRose (jderose) wrote :

Okay, after more thought and research, here is my proposal for what to do about this in Precise, particularly considering that UbuntuOne must drop CouchDB sync:

https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/903098

Kate Stewart (kate.stewart) wrote :

We're gettting pretty late in Precise, is this going to be deferred for Q?

Changed in couchdb (Ubuntu Precise):
assignee: nobody → Ubuntu One Desktop+ team (ubuntuone-desktop+)
Jason Gerard DeRose (jderose) wrote :

Kate, I just can't afford any more time on this bug personally, so looks like this wont happen for Precise.

Although this isn't really a regression as the current package in Precise is broken in the same way. The packaging stuff I could figure out, but I don't think I have the expertise needed with Erlang and CouchDB internals to fix the /etc/init.d script.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package couchdb - 1.2.0-2ubuntu1

---------------
couchdb (1.2.0-2ubuntu1) quantal; urgency=low

  * Merge from Debian unstable (LP: #1022515, LP: #817656):
    - fixes ownership of /etc/couchdb. LP: #731272.
  * Remaining changes:
    - Use pkg-config instead of the js-config script shipped by libmozjs
    - debian/rules, debian/control: split couchdb and couchdb-bin
    - debian/postinst: renamed to couchdb-bin.postinst
    - debian/couchdb-bin.postrm: don't try to delete couchdb system
      user/group
  * Dropped changes, superseded upstream:
    - Backport svn r1039345 from 1.0.3
    - debian/patches/couchio-*: patchset from CouchIO to fix U1 replication
      over SSL. It's not clear if these were upstreamed, but most of the
      source files they applied to no longer exist at all upstream, so if
      these are still needed they will need to be redone.
    - debian/patches/fix-help2man.patch: dropped, no longer needed
    - debian/patches/jquery15-fix.patch: Make Futon work with jQuery 1.5
    - debian/patches/moz*.patch: Spidermonkey 1.8.54 patchset
  * Dropped changes, included in Debian:
    - debian/control - bump standards version
    - debian/control - fix lintian error not-binnmuable-all-depends-any
    - Build against a proper libmozjs
    - Update build-depends/depends/build config (we no longer need the
      --with-js build flags)
  * Dropped changes, superseded in Debian:
    - source/format - mark package as 1.0 format
    - change the default permissions on /etc/couchdb to 0775 and files to
      0664

  [ Jason Gerard DeRose ]
  * /etc/couchdb/default.ini and /etc/couchdb/default.d are delivered in the
    `couchdb-bin` package and are owned by root
  * /etc/couchdb/local.ini and /etc/couchdb/local.d are delivered in the
  * The `couchdb-bin` package does not create nor require the "couchdb" user
    (this is now done in the `couchdb` package instead). LP: #1007125.
  * Added a short sleep delay in couchdb.postrm so couchdb is more likely to
    have actually terminated by the time we `deluser couchdb`, which is needed
    for `sudo apt-get purge couchdb` to work when couchdb is running
 -- Steve Langasek <email address hidden> Tue, 31 Jul 2012 02:04:48 +0000

Changed in couchdb (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers