Disposable prototype of community import

Bug #347498 reported by Paul Everitt on 2009-03-23
2
Affects Status Importance Assigned to Milestone
KARL3
Medium
claytron

Bug Description

Spend a couple of hours looking at the bulk importer and trying to get a new script that imports communities from the GenericSetup sample data.

Steps
======

1) Write a console script that is an initial step for a KARL2 importer.

2) Have it bootstrap itself similar to the bulkloader.py

3) Point it at the sample data and produce a sequence of Community objects.

Background
=============

Early in the project we wanted to do some architectural studying of performance with large data sets. We wrote a bulk loader against some artificially randomized sample data. The bulk loader ran as a ZEO client console script. It lives in src/osi/osi/scripts/bulkloader.py and can be run as follows:

$ cd src/osi
$ cd ../..

(Those make sure you start from the correct directory)

$ bin/loadcontent --repetitions 3 --filedir src/osi/osi/scripts/sampledata/ src/osi/osi/scripts/sampledata/small-generated.xml.gz

Unfortunately the code in there is a bit out of date, so it's only a starting point. To see, for example, the things needed to correctly create a community, look in src/karl/karl/views/communities.py, in def add_community_view.

About Console Scripts
===================

Console scripts are easy to write, once you get used to them:

http://carlos.agendaless.com/dko/tutorials/t2-contentandcataloging/feedprocessing.html#making-a-console-script

You can see the entry point in src/osi/setup.py for the loadcontent console script. Whenever you edit setup.py, make sure to run buildout again.

Changed in karl3:
assignee: nobody → clayton-sixfeetup
importance: Undecided → Medium
milestone: none → m7
claytron (clayton-sixfeetup) wrote :

basic console script has been setup. need to get a test GenericSetup dump from a Karl 2 site (#347474) so that we can test with 'real' data.

Changed in karl3:
status: New → In Progress
Paul Everitt (paul-agendaless) wrote :

Triage to M8.

Changed in karl3:
milestone: m7 → m8
claytron (clayton-sixfeetup) wrote :

Karl2 community objects are now being imported by the importer script

Changed in karl3:
status: In Progress → Fix Committed
Changed in karl3:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers