Comment 5 for bug 622194

On 23 August 2010 01:15, Mark Hattam <email address hidden> wrote:
> If the max_allowed_packet is smaller than the gedcom filesize it fails,
> MySQL doesn't attempt to chunk the gedcom file to get it into the table
> in packets that are "allowed".

"Chunking" is the way to go. The original version of the importer broke the
file in to 64KB chunks.

The problem was that if multi-byte characters get split, then all sorts of nasty
things happen, and we don't know if the file is UTF8, ANSEL or whatever
until afterwards. Dealing with this can be done, but it slowed the import,
and caused lots of timeout problems.

The current "load it all" technique avoids timeout problems because all
the work is handled in a single SQL statement, and (on UNIX) systems,
time spent in SQL does not count towards the execution time limit.

Greg
--
Greg Roach, Stony Stratford, Bucks, England, <email address hidden>
"Don't eat animals - it's not good for them and they don't like it"