times.dbm incompatible based on Python version it was created with

[clayg]

On my deadsnakes [1] python2.6I get can't get my py26 environment to run.

    py26 runtests: commands[0] | python setup.py testr --testr-args=
    running testr
    running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests
    No module named _bsddb
    error: testr failed (3)
    ERROR: InvocationError: '/home/clayg/.python-swiftclient/.tox/py26/bin/python setup.py testr --testr-args='
    ERROR: py26: commands failed

What seems to be going on is just that using anydbm with the stock system python (py27) is getting dbhash and it works just fine, then when the py26 env loops around and uses anydbm it inspects the db and tries to open it with bsddb even though it's anydbm._defaultmod is dmb (or gdbm w/e depending on the ImportError). I mean it's easy to say deadsnakes is just broken, but it's sort of a great way to get you some python2.6 on an ubuntu system I wonder if you might just work around it by avoiding bsddb explicitly?

I mean, it's easy to get wrong, even macosx has a broken bsddb [2]:

    clayg:~/Workspace/swift$ python -c 'import sys; print sys.version; import bsddb'
    2.7.2 (default, Oct 11 2012, 20:14:37)
    [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)]
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bsddb/__init__.py", line 67, in <module>
        import _bsddb
    ImportError: No module named _bsddb

I tried to compile python2.6 with myself with "bsddb185 bsddbmodule.c", but it's kind of a chore to get the bsddb-dev 4.0 includes these days cause everything is deprecated :\

Don't tell anyone, but what I ended up doing was just hacking my local system anydb._names to just not look at hashdb, I think other folks just hack hashdb to use bsddb3 [3]

1. https://launchpad.net/~fkrull/+archive/deadsnakes
2. http://stackoverflow.com/questions/814041/how-to-fix-the-broken-bsddb-install-in-the-default-python-package-on-mac-os-x-10
3. http://marc-abramowitz.com/archives/2007/11/28/hacking-os-xs-python-dbhash-and-bsddb-modules-to-work/

[Robert Collins (lifeless)]

You can get a backtrace with
TESTR_PDB=1 testr run

[Robert Collins (lifeless)]

That, said, there is also bug 1229445 and I think this is arguably a dupe.

Could you check on your system, does
dbm (in python 2.x)
and dbm.ndbm (in python3.x)
work?

Thanks!

[John Garbutt (johngarbutt)]

I was seeing this same error, but in a different way, and managed to fix it up. Basically default python27 in the mac seems broken.

On my mac, I saw this because I used port to install python27 but not virtualenv, installing the port version of virtualenv brought the fix into my virtualenv when running ./run-tests.sh

Interestingly tox correctly picked up the port install of python27, so never showed this issue.

[Robert Collins (lifeless)]

So, I don't have a mac - I really need guidance on where to take this :(

[clayg (clay-gerrard)]

well, i ran into this again in a different context coming at it from https://bugs.launchpad.net/testrepository/+bug/1229445 this time... In all fairness I wonder if some of this really should just be filed against openstack - since we're the ones making all the noise.

I think maybe instead of anydbm if testrepository/repository/file.py explicitly chose a specific format it thinks is going to be well supported across various builds of 2.6 - 3.4 when creating a db it might have a better time of it.

For existing db's I guess you just open it with anydb and get whatever is in there. But when making the db the first time (or after someone deleted it trying to dig themselves out of this mess) it seems like just picking whatever python's happen to be running first's default isn't always the most widely supported choice.

OTOH, maybe tox should be isolating this better? Is there a good reason to support sharing this db across runs of different virtualenv's with different python interpreters?

my launchpad says i'm subscribed to this bug - so you if you need any more information or debugging from my end - maybe I'll see the next update.

[Robert Collins (lifeless)]

So the problem is that I tried that. I couldn't find a db format supported across Windows MacOSX Linux 2.x and 3.x when I last looked into this.

I would *totally* accept a patch to default to such a format if there is one.

Some prior context, but sadly I can't find the prior bug about this specific incompat issue.

See https://bugs.launchpad.net/testrepository/+bug/1091500
https://bugs.launchpad.net/testrepository/+bug/1094330

[Morgan Fainberg (mdrnstm)]

Can we simply make the timing db (and the output) python-version specific? e.g. py2.7 gets a py27 dir, a py3.4 gets a py34 dir. It doesn't have to work across platforms just isolate per python-version.