Metadata Import slower in 2.21

Bug #1536817 reported by Jim Grace on 2016-01-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
DHIS
High
Morten Olav Hansen

Bug Description

A metadata import of 1090 data elements (with user group sharing information) that took 946 seconds in v2.20, took 2,932 seconds in v2.21.

I can supply the import to developers.

Thanks for reporting this one Jim, I'm seeing the same thing, about a 3 to
4x increase in metaData importation time with 2.21 over 2.20.

*Timothy Harding*
Sr. Systems Analyst, BAO Systems
+1 202-536-1541 | <email address hidden> | http://www.baosystems.com | Skype:
<email address hidden> | 2900 K Street, Suite 404, Washington D.C. 20007

On Fri, Jan 22, 2016 at 7:37 AM, Jim Grace <email address hidden>
wrote:

> Public bug reported:
>
> A metadata import of 1090 data elements (with user group sharing
> information) that took 946 seconds in v2.20, took 2,932 seconds in
> v2.21.
>
> I can supply the import to developers.
>
> ** Affects: dhis2
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are a member of DHIS 2
> developers, which is subscribed to DHIS.
> https://bugs.launchpad.net/bugs/1536817
>
> Title:
> Metadata Import slower in 2.21
>
> Status in DHIS:
> New
>
> Bug description:
> A metadata import of 1090 data elements (with user group sharing
> information) that took 946 seconds in v2.20, took 2,932 seconds in
> v2.21.
>
> I can supply the import to developers.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>

Changed in dhis2:
milestone: none → 2.23
assignee: nobody → Morten Olav Hansen (mortenoh)
importance: Undecided → Medium
importance: Medium → High
Morten Olav Hansen (mortenoh) wrote :

Hi Timothy

Was that with or without preheating? the default is now off. We are
rewriting the importer for 2.23, so expect a more robust (and hopefully a
lot quicker) importer there.

Would it be possible to test without the sharing objects also? just to see
if the results are the same.

Please also know that 2.21, 2.22 release both add more validations.. so a
bit extra time is to be expected (but shouldn't be a 3-4x increase)

--
Morten

On Wed, Jan 27, 2016 at 2:54 PM, Timothy Harding <email address hidden>
wrote:

> Thanks for reporting this one Jim, I'm seeing the same thing, about a 3 to
> 4x increase in metaData importation time with 2.21 over 2.20.
>
>
> *Timothy Harding*
> Sr. Systems Analyst, BAO Systems
> +1 202-536-1541 | <email address hidden> | http://www.baosystems.com | Skype:
> <email address hidden> | 2900 K Street, Suite 404, Washington D.C. 20007
>
> On Fri, Jan 22, 2016 at 7:37 AM, Jim Grace <email address hidden>
> wrote:
>
>> Public bug reported:
>>
>> A metadata import of 1090 data elements (with user group sharing
>> information) that took 946 seconds in v2.20, took 2,932 seconds in
>> v2.21.
>>
>> I can supply the import to developers.
>>
>> ** Affects: dhis2
>> Importance: Undecided
>> Status: New
>>
>> --
>> You received this bug notification because you are a member of DHIS 2
>> developers, which is subscribed to DHIS.
>> https://bugs.launchpad.net/bugs/1536817
>>
>> Title:
>> Metadata Import slower in 2.21
>>
>> Status in DHIS:
>> New
>>
>> Bug description:
>> A metadata import of 1090 data elements (with user group sharing
>> information) that took 946 seconds in v2.20, took 2,932 seconds in
>> v2.21.
>>
>> I can supply the import to developers.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : <email address hidden>
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help : https://help.launchpad.net/ListHelp
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>

We had unintentionally set cache preheat to off for meta-data importer. We have set this back to on in latest 2.22 and trunk. Could you please try again now?

Morten Olav Hansen (mortenoh) wrote :

A few things caused this, preheat being false on 2.21, attribute values being validated even there was nothing to validate etc.

Please note that performance on some imports will be slower going forward as we are doing a lot more validations than before.

Also note that I am rewriting the importer completely, so hopefully there will be performance upgrades related to that also.

For now, update to latest 2.21, 2.22, and 2.23 for what I could fix.

Changed in dhis2:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers