Disposable prototype of community import

[Paul Everitt]

Spend a couple of hours looking at the bulk importer and trying to get a new script that imports communities from the GenericSetup sample data.

Steps
======

1) Write a console script that is an initial step for a KARL2 importer.

2) Have it bootstrap itself similar to the bulkloader.py

3) Point it at the sample data and produce a sequence of Community objects.

Background
=============

Early in the project we wanted to do some architectural studying of performance with large data sets. We wrote a bulk loader against some artificially randomized sample data. The bulk loader ran as a ZEO client console script. It lives in src/osi/osi/scripts/bulkloader.py and can be run as follows:

$ cd src/osi
$ cd ../..

(Those make sure you start from the correct directory)

$ bin/loadcontent --repetitions 3 --filedir src/osi/osi/scripts/sampledata/ src/osi/osi/scripts/sampledata/small-generated.xml.gz

Unfortunately the code in there is a bit out of date, so it's only a starting point. To see, for example, the things needed to correctly create a community, look in src/karl/karl/views/communities.py, in def add_community_view.

About Console Scripts
===================

Console scripts are easy to write, once you get used to them:

http://carlos.agendaless.com/dko/tutorials/t2-contentandcataloging/feedprocessing.html#making-a-console-script

You can see the entry point in src/osi/setup.py for the loadcontent console script. Whenever you edit setup.py, make sure to run buildout again.

[claytron (clayton-sixfeetup)]

basic console script has been setup. need to get a test GenericSetup dump from a Karl 2 site (#347474) so that we can test with 'real' data.

[Paul Everitt (paul-agendaless)]

Triage to M8.

[claytron (clayton-sixfeetup)]

Karl2 community objects are now being imported by the importer script