hockeypuck

Support PostgreSQL bulk loading techniques

Bug #1208295 reported by Casey Marshall on 2013-08-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
​ hockeypuck
Fix Committed
Medium
Casey Marshall
hockeypuck 1.0 "bitquantum of solace"
Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

Loading an SKS dump into Hockeypuck+PostgreSQL has some problems -- insert performance takes a dive -- from 60k keys/hr down to 1-4k/hr as the size grew from 0-500k keys.

The OpenPGP schema is full of foreign key references, other constraints and indexes. Could support disabling/enabling these, perhaps using a special loader utility. Or, direct offline loading with COPY.

http://www.postgresql.org/docs/current/interactive/populate.html

Casey Marshall (cmars) on 2013-08-05
Changed in hockeypuck:
milestone: none → 1.0
Casey Marshall (cmars) wrote :

Creating a utility that inserts directly from SKS dumps with dropped constraints to build an initial database.

Changed in hockeypuck:
status: Triaged → In Progress
Casey Marshall (cmars) wrote :

Need to add support for rebuilding the ptree database as well.

Casey Marshall (cmars) wrote :

hockeypuck-import now inserts in large transactions with dropped constraints, then de-duplicates and adds constraints.

Changed in hockeypuck:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.
Subscribing...

Other bug subscribers