Produce Regular Data Dumps - full & incremental

[Aaron Swartz]

We need regular dumps of our database. Many people have been complaining about this.

[Anand Chitipothu (anandology)]

Are you talking about the dump.txt text file in the repository?

[Aaron Swartz (aaronsw)]

> Are you talking about the dump.txt text file in the repository?

No, people want dumps of all the data we've imported, not really the templates.

[Aaron Swartz (aaronsw)]

Assigning this to Edward. Edward, can you make sure some tdb log files or something like that are available as a result of the next import you do?

[Anand Chitipothu (anandology)]

I have added an action to dump all the data, but I was talking far too much time.
I think, we also need to generate a fresh tdb.log for the entire database once. I'll file a new bug for that.

[Anand Chitipothu (anandology)]

jsondumps for all editions and authors are generated and kept here: http://openlibrary.org/static/jsondump

Need to setup cron to generate them every week.

[solrize (solrize)]

Should also include books.json.gz (the expanded dump).

[Edward Betts (edwardbetts)]

We should make this run daily.

[George (george-archive)]

2010-03-29
---------------

I'm planning to generate 3 types of dumps with OL data.

Open Library Dump:
     description: Latest revisions of all documents
     filename: ol_dump_${date}.txt.gz
     columns: key, type, revision, json
     frequency: monthly
     sort-order: unspecified

Open Library Complete Dump:
     description: All revisions of all documents
     filename: ol_cdump_${date}.txt.gz
     columns: key, type, revision, json
     frequency: monthly
     sort-order: unspecified

Open Library Incremental Dump:
     description: All revisions of all documents modified in a given day
     filename: ol_idump_${date}.txt.gz
     columns: key, type, revision, json
     frequency: daily
     sort-order: modification time

Each of these dump will be stored as an item in the internet archive
cluster.

URL Format:

Even though these files are stored in IA, there will be a
openlibrary.org/* URL for each file.

I'm considering the following two url formats.

Option#1:
http://openlibrary.org/dumps/ol_dump_2010-03-31.txt.gz
http://openlibrary.org/dumps/ol_cdump_2010-03-31.txt.gz
http://openlibrary.org/dumps/ol_idump_2010-03-31.txt.gz

Option#2:
http://openlibrary.org/dumps/2010/03/ol_dump_2010-03-31.txt.gz
http://openlibrary.org/dumps/2010/03/ol_cdump_2010-03-31.txt.gz
http://openlibrary.org/dumps/2010/03/ol_idump_2010-03-31.txt.gz

Which one should we pick?

Any other suggestions/feedback?

Anand

[George (george-archive)]

Suggested location - http://openlibrary.org/data/ *

Also - are you suggesting that you'll stop producing edition/author dumps separately? Or a Works dump of some sort, separately?

[George (george-archive)]

Brewster asks: how about being less database-like and use our JSON logs instead?
Periodically, compress the JSON log and put it into an archive item.

[bradymatthews (brady-matthews)]

Any updates on when a dump might happen?

[George (george-archive)]

Anand?

[Anand Chitipothu (anandology)]

Will be ready by end of this week.

[Anand Chitipothu (anandology)]

Latest dump is available at http://www.archive.org/details/ol_dump_2010-04-30

[bradymatthews (brady-matthews)]

Thanks! :D

[bradymatthews (brady-matthews)]

I'm looking through the works data, and I'm seeing many, many entries for Tom Sawyer. Is there an identifier in there somewhere that I can use to only get the original, or is that something I'll need to write a bot to try to figure out?

[Anand Chitipothu (anandology)]

Found a bug in the dump generation. Will regenerate a new dump shortly.

[Anand Chitipothu (anandology)]

Regenerated the dumps and uploaded them to http://www.archive.org/details/ol_dump_2010-04-30.

[George (george-archive)]

This is finished now, right?

Or, do you still publish the dumps manually?

[George (george-archive)]

Anand - is this done now?

[George (george-archive)]

Anand - we were under the impression (perhaps incorrectly) that you are making daily dumps. If that's not right, can you confirm for us what's happening?

[Anand Chitipothu (anandology)]

On 01-Jul-10, at 3:26 AM, George wrote:

> Anand - we were under the impression (perhaps incorrectly) that you
> are
> making daily dumps. If that's not right, can you confirm for us what's
> happening?

No, daily dumps is not happening right now. Only monthly dumps are
happening.

http://www.archive.org/details/ol_exports

[Anand Chitipothu (anandology)]

Uploading the dumps automatically to archive.org is take care now.

Generation of montly dumps is completely in place. Need to work on generating daily incremental dumps. Reducing the priority as that is not very critical.

[George (george-archive)]

Great stuff, Anand. Agreed about the priority.