combining ROOT files in fixed-order mode

Bug #1758683 reported by Oleksandr Zenaiev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MadGraph5_aMC@NLO
In Progress
Undecided
Olivier Mattelaer

Bug Description

Dear authors,
what I am going to report is not a bug but maybe a feature which can be improved.

I am generating a process at NLO such that the last calculation is splitted into about 1500 jobs, and I am filling about 50000 histograms such that each resulting ROOT file has size of 30 MB. In the end of run, combining them in the standard way via combine_root.C has following difficulties:
(1) it takes about 12 hours and uses about 100 GB of RAM, so it is pretty not obvious to succeed
(2) it even does not work out-of-the-box because of the hardcoded limit of 1000 max. files in combine_root.C (of course it was easy to change).

Also I've tried the standard 'hadd' ROOT command. It seems to perform much more efficient and gives the identical combined file: with same input it takes ~ 30 minutes and uses 20 GB only.
I am using MG5_aMC_v2_6_0, gcc 5.3 and ROOT 5.34.

Is there a reason to use the custom script (combine_root.C) for combining output ROOT files? What do you think, would it make sense to switch to the standard ROOT tool 'hadd' instead?

Best regards,
Sasha.

Changed in mg5amcnlo:
assignee: nobody → Stefano Frixione (stefano-frixione)
Revision history for this message
Stefano Frixione (stefano-frixione) wrote :

Dear Sasha,
thanks for reporting this.
Admittedly, that part of the package was never meant to be rock solid, and hasn't been
tested with different root versions, or with large files. Thus, in this case it is not
suprising that it doesn't work well.
In spite of being the original author of that piece of code, I can't remember why hadd
was not deemed a viable option. Maybe it was simply due to my poor understanding of root.
I'll investigate as soon as I'll have some time. In the meanwhile, I have nothing against
you using hadd, but I strongly suggest to test its results against those of the standard
package in a few cases where the latter is supposed to do well, and which are quick to run.
Best, Stefano.

Changed in mg5amcnlo:
status: New → In Progress
Revision history for this message
Olivier Mattelaer (olivier-mattelaer) wrote :

Hi,

I will try to move forward on this.
Would you have a patch (or an example of your changes?)
I have never used root so if you can detail what you did to fix the issue it could be great (otherwise I will wait that someone provide us such clever way of combining histograms)

Cheers and thanks,

Olivier

Changed in mg5amcnlo:
assignee: Stefano Frixione (stefano-frixione) → Olivier Mattelaer (olivier-mattelaer)
Revision history for this message
Oleksandr Zenaiev (zenaiev) wrote :

Hi Olivier,
I did some modification of Template/NLO/SubProcesses/combine_root.sh, please find attached the diff file.
Let me know if you will have further questions.
Cheers,
Sasha.

Revision history for this message
Oleksandr Zenaiev (zenaiev) wrote :

Forgot to mention: this diff is created using MG5_aMC_v2_6_0

Revision history for this message
Olivier Mattelaer (olivier-mattelaer) wrote :

Thanks so much for this.

We are reviewing it and will include those changes in the next version.

Thanks,

Olivier

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.