Pythi8 crashes while merging split runs

[Christopher Verhaaren]

This does not happen every time I run Pythia8, but in this process pythia8 crashes after beginning to merge results.

Merging results from the split PY8 runs...
INFO: Create matching plots for Pythia8
Command "pythia8" interrupted with error:
IndexError : list index out of range

error log is attached.

[Valentin Hirschi (valentin-hirschi)]

An issue occurred when MG5_aMC tried to read the histograms generated by PY8.

You are doing an MLM-merging simulation and the PY8 outputs the so-called DJR plots and Pt distribution of jets of various subleading orders, which you can use to diagnose the quality of the merging.

The error occurred here when trying to read the raw plot data:

in <MG5_aMC_root_path>/madgraph/interface/common_run_interface.py, around line ~1130

---
            min_scale_histos = histograms.HwUList(data_path,
                               consider_reweights=[], run_id=0,
                                        merging_scale=min_merging_scale)
            max_scale_histos = histograms.HwUList(data_path,
                               consider_reweights=[], run_id=0,
                                        merging_scale=max_merging_scale)
---

please re-run the Pythia8 shower after having modified the file above as follows:

----
            misc.sprint("I am trying to read PY8 plots from:", data_path)
            min_scale_histos = histograms.HwUList(data_path,
                               consider_reweights=[], run_id=0,
                                        merging_scale=min_merging_scale)
            max_scale_histos = histograms.HwUList(data_path,
                               consider_reweights=[], run_id=0,
                                        merging_scale=max_merging_scale)
            misc.sprint("I obtained that many plots", len(min_scale_histos), len(max_scale_histos))
----

You should re-run Pythia8 like this:

cd <MG5_aMC_root_path>
./bin/mg5_aMC
MG5_aMC>launch proc_sm_ee_bbbbvv_QCD_0_hhvv -i
proc_sm_ee_bbbbvv_QCD_0_hhvv > shower pythia8 run_<XXX> -f
[...]
It should crash again, this time printing out the diagnostic lines added above. Please report what these are. In particular, this should tell you what is the path of the raw data source it is trying to read, send me those raw data plot file by email directly at <email address hidden>

And <XXX> is the ID of the run that actually failed in your simulation (so run_01 if it was the first one).

Finally, please also tell me if Pythia8's run also crash when turning off parallelization. You can do so simply by running the following command before 'shower pythia8 run_<XXX> -f':

proc_sm_ee_bbbbvv_QCD_0_hhvv > set nb_core 1

[Christopher Verhaaren (cver)]

Okay, I added the code to common_run_interface.py, but there was no change
to the output, that is there were no diagnostic out lines. While I saved
the python file I did nothing else. Do I need to recompile it or something
like that?

When I run with just one core (set nb_core 1) I get the same error:

INFO: Create matching plots for Pythia8

*Command "shower pythia8 sm_ee_bbbbvv_QCD_0_hhvv_1TeV_50k_pyde -f"
interrupted with error:*

*IndexError : list index out of range*

I should mention that I have used the parallel pythia8 for other processes
and everything has worked fine.

I've attached the log from the last run, in case it is useful. However, it
seems that my changing the python code did not affect anything.

Thanks for the help,

On Mon, Jul 24, 2017 at 2:59 PM, Valentin Hirschi <
<email address hidden>> wrote:

> An issue occurred when MG5_aMC tried to read the histograms generated by
> PY8.
>
> You are doing an MLM-merging simulation and the PY8 outputs the so-
> called DJR plots and Pt distribution of jets of various subleading
> orders, which you can use to diagnose the quality of the merging.
>
> The error occurred here when trying to read the raw plot data:
>
> in <MG5_aMC_root_path>/madgraph/interface/common_run_interface.py,
> around line ~1130
>
>
> ---
> min_scale_histos = histograms.HwUList(data_path,
> consider_reweights=[], run_id=0,
> merging_scale=min_merging_scale)
> max_scale_histos = histograms.HwUList(data_path,
> consider_reweights=[], run_id=0,
> merging_scale=max_merging_scale)
> ---
>
> please re-run the Pythia8 shower after having modified the file above as
> follows:
>
> ----
> misc.sprint("I am trying to read PY8 plots from:", data_path)
> min_scale_histos = histograms.HwUList(data_path,
> consider_reweights=[], run_id=0,
> merging_scale=min_merging_scale)
> max_scale_histos = histograms.HwUList(data_path,
> consider_reweights=[], run_id=0,
> merging_scale=max_merging_scale)
> misc.sprint("I obtained that many plots",
> len(min_scale_histos), len(max_scale_histos))
> ----
>
> You should re-run Pythia8 like this:
>
> cd <MG5_aMC_root_path>
> ./bin/mg5_aMC
> MG5_aMC>launch proc_sm_ee_bbbbvv_QCD_0_hhvv -i
> proc_sm_ee_bbbbvv_QCD_0_hhvv > shower pythia8 run_<XXX> -f
> [...]
> It should crash again, this time printing out the diagnostic lines added
> above. Please report what these are. In particular, this should tell you
> what is the path of the raw data source it is trying to read, send me those
> raw data plot file by email directly at valentin.hirschi@
> REMOVETHISgmail.com
>
> And <XXX> is the ID of the run that actually failed in your simulation
> (so run_01 if it was the first one).
>
> Finally, please also tell me if Pythia8's run also crash when turning
> off parallelization. You can do so simply by running the following
> command before 'shower pyth...

[Valentin Hirschi (valentin-hirschi)]

So the issue was indeed a bug in the code.

Around line 1165 of <MG5_aMC_root_dir>/interface/common_interface.py, change:

            histograms.HwUList(min_scale_histos+max_scale_histos).output(
                pjoin(plot_root_path,'min_max_%s_%s_comparison'
                                         %(merging_scale_name,observable_name)),
                format='gnuplot',
                uncertainties=[],
                ratio_correlations=True,
                arg_string='Automatic plotting from MG5aMC',
                jet_samples_to_keep=[],
                use_band=[],
                auto_open=False)
        return True

into

            histograms.HwUList(min_scale_histos+max_scale_histos).output(
                pjoin(plot_root_path,'min_max_%s_%s_comparison'
                                         %(merging_scale_name,observable_name)),
                format='gnuplot',
                uncertainties=[],
                ratio_correlations=True,
                arg_string='Automatic plotting from MG5aMC',
                jet_samples_to_keep=None,
                use_band=[],
                auto_open=False)
        return True

that is

jet_samples_to_keep=[] -> jet_samples_to_keep=None

This fix is now pushed to the development version and will be part of the next release.

But feel free to already fix your local version in the above way.

[Christopher Verhaaren (cver)]

Thanks for the fix!

On Mon, Jul 24, 2017 at 4:51 PM, Valentin Hirschi <
<email address hidden>> wrote:

> So the issue was indeed a bug in the code.
>
> Around line 1165 of <MG5_aMC_root_dir>/interface/common_interface.py,
> change:
>
> histograms.HwUList(min_scale_histos+max_scale_histos).output(
> pjoin(plot_root_path,'min_max_%s_%s_comparison'
> %(merging_scale_name,
> observable_name)),
> format='gnuplot',
> uncertainties=[],
> ratio_correlations=True,
> arg_string='Automatic plotting from MG5aMC',
> jet_samples_to_keep=[],
> use_band=[],
> auto_open=False)
> return True
>
> into
>
> histograms.HwUList(min_scale_histos+max_scale_histos).output(
> pjoin(plot_root_path,'min_max_%s_%s_comparison'
> %(merging_scale_name,
> observable_name)),
> format='gnuplot',
> uncertainties=[],
> ratio_correlations=True,
> arg_string='Automatic plotting from MG5aMC',
> jet_samples_to_keep=None,
> use_band=[],
> auto_open=False)
> return True
>
> that is
>
> jet_samples_to_keep=[] -> jet_samples_to_keep=None
>
> This fix is now pushed to the development version and will be part of
> the next release.
>
> But feel free to already fix your local version in the above way.
>
> ** Changed in: mg5amcnlo
> Status: New => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1706139
>
> Title:
> Pythi8 crashes while merging split runs
>
> Status in MadGraph5_aMC@NLO:
> Fix Committed
>
> Bug description:
> This does not happen every time I run Pythia8, but in this process
> pythia8 crashes after beginning to merge results.
>
> Merging results from the split PY8 runs...
> INFO: Create matching plots for Pythia8
> Command "pythia8" interrupted with error:
> IndexError : list index out of range
>
> error log is attached.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mg5amcnlo/+bug/1706139/+subscriptions
>