marc_export - performance issues with larger numbers of copies per bib
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Evergreen |
Fix Released
|
Medium
|
Unassigned | ||
3.10 |
Won't Fix
|
Medium
|
Unassigned | ||
3.11 |
Fix Released
|
Medium
|
Unassigned |
Bug Description
As reported by Jason Stephenson, marc_export was taking a long time to export data in the case of including items (--items) and with bibs that had a large number of copies. (Hundreds to thousands of copies per bib).
Testing on EG 3.11.1
libmarc-record-perl 2.0.7-1
On a test system, I identified our top 100 or so bibs with the most copies and duplicated the copies on each bib 4 times, so went from 8K to 120K copies so I could trigger the issue. The bibs with the most number of copies had 16K, 6K, 4K and 3K copies each.
I used libdevel-
The problem is that the code that adds each 852 tag uses $marc->
https:/
By re-working how the 852s get added, by using insert_
I'm not sure how much the memory usage has gone up yet, I'll check that next.
The xml output was identical also, so I think I haven't changed the ordering at all.
I ran this with just the list of bibs with the lots of copies with the --pipe option.
Example command line runs with profiling.
## EG 3.11 version
$ cat ids-most-items.txt | perl -d:Profile /openils/
time elapsed (wall): 820.5668
time running program: 609.7940 (74.31%)
time profiling (est.): 210.7729 (25.69%)
number of calls: 286352387
number of exceptions: 1
%Time Sec. #calls sec/call F name
46.07 280.9194 122512 0.002293 MARC::Record:
23.84 145.3557 234275686 0.000001 MARC::Field::tag
7.46 45.5201 122512 0.000372 MARC::Record:
7.12 43.4227 105 0.413549 Marque:
6.32 38.5100 38895909 0.000001 <anon>:
1.78 10.8453 123770 0.000088 Fieldmapper:
1.25 7.6325 122722 0.000062 MARC::Record:
1.22 7.4176 106 0.069977 Marque:
$ cat ids-most-items.txt | perl -d:Profile ./marc_
time elapsed (wall): 165.3634
time running program: 129.9096 (78.56%)
time profiling (est.): 35.4539 (21.44%)
number of calls: 51348206
number of exceptions: 1
%Time Sec. #calls sec/call F name
32.55 42.2917 105 0.402778 Marque:
31.70 41.1810 38895909 0.000001 <anon>:
8.66 11.2528 123770 0.000091 Fieldmapper:
5.01 6.5075 106 0.061391 Marque:
3.39 4.4023 105 0.041927 MARC::File:
2.14 2.7745 2427108 0.000001 MARC::File:
2.09 2.7136 123770 0.000022 Fieldmapper:
2.02 2.6192 2089846 0.000001 <anon>:
I'll put my full testing notes at
https:/
And a working branch will be on the way.
Josh
tags: | added: cat-importexport |
Changed in evergreen: | |
milestone: | none → 3.12-beta |
assignee: | Jason Stephenson (jstephenson) → nobody |
tags: | added: pullrequest |
Changed in evergreen: | |
assignee: | nobody → Jane Sandberg (sandbergja) |
Changed in evergreen: | |
status: | Confirmed → Fix Committed |
tags: | added: signedoff |
Changed in evergreen: | |
assignee: | Jane Sandberg (sandbergja) → nobody |
Changed in evergreen: | |
status: | Fix Committed → Fix Released |
The memory usage increase seems slight, less than 1%.
(kbytes): 227200
to
(kbytes): 229004
= 0.794014% increase