Change in production xsecs in SUSY models in 2.6.0

[Zachary Marshall]

Hi there,

I'm hoping that you can help us track this issue down, as it's holding up our validation of MadGraph 2.6.0. Emma Kuwertz and I found that between older and newer MadGraph releases, the cross section of p p > go go j processes changed by 50% or more. I set up and run out of the box:

import model mssm
generate p p > go go
add process p p > go go j
output -f

in MadGraph 2.3.3 and 2.4.3. I run the same in MadGraph 2.6.0, but with `import model MSSM_SLHA2`. In both releases I then set the masses of all SUSY particles to be 10**9 GeV, except the gluino which is set to 1 TeV. All other parameters I leave as they are, out of the box, but a quick check suggests that there aren't wild new cuts that have appeared. With 2.3.3 the cross section I get is:

     Cross-section : 0.662 +- 0.00212 pb

With 2.4.3 and 2.6.0, I get the same answer:

     Cross-section : 0.7888 +- 0.002181 pb

There, the cross section files suggest that most of the difference is due to a large (>25%) change in g g > go go g. I don't see any difference in the PDFs that might cause such a change, though.

In Emma's run inside the ATLAS framework, we were looking at more modest total cross section changes, but large changes to the go go j cross section (of order 50%) that led to significantly harder kinematics in one case than the other.

Is there some known feature of the releases that would have caused these differences? I tried to look over the update notes, but I didn't find anything sinister.

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

Looks like that the pdf reweighting was not performed anymore.
If you were running with pdfwgt = F you will see that both were in agreement (like CMS is running if I’m correct).
This clearly indicates that you are sensitive to scale variation since such changes are related to the PDF scale.
(so in itself this is not a catastrophic bug to my point of view)

I have created a patch that should put him back.
(see below) I will run some tests with some other process to check that this is not introducing some other problem.
After that fix, I observes that for the gluon channel, I indeed have an error of 50% due to scale uncertainty (so this seems coherent).

Cheers,

Olivier

> On Nov 9, 2017, at 02:33, Zachary Marshall <email address hidden> wrote:1
> Public bug reported:
>
> Hi there,
>
> I'm hoping that you can help us track this issue down, as it's holding
> up our validation of MadGraph 2.6.0. Emma Kuwertz and I found that
> between older and newer MadGraph releases, the cross section of p p > go
> go j processes changed by 50% or more. I set up and run out of the box:
>
> import model mssm
> generate p p > go go
> add process p p > go go j
> output -f
>
> in MadGraph 2.3.3 and 2.4.3. I run the same in MadGraph 2.6.0, but with
> `import model MSSM_SLHA2`. In both releases I then set the masses of
> all SUSY particles to be 10**9 GeV, except the gluino which is set to 1
> TeV. All other parameters I leave as they are, out of the box, but a
> quick check suggests that there aren't wild new cuts that have appeared.
> With 2.3.3 the cross section I get is:
>
> Cross-section : 0.662 +- 0.00212 pb
>
> With 2.4.3 and 2.6.0, I get the same answer:
>
> Cross-section : 0.7888 +- 0.002181 pb
>
> There, the cross section files suggest that most of the difference is
> due to a large (>25%) change in g g > go go g. I don't see any
> difference in the PDFs that might cause such a change, though.
>
> In Emma's run inside the ATLAS framework, we were looking at more modest
> total cross section changes, but large changes to the go go j cross
> section (of order 50%) that led to significantly harder kinematics in
> one case than the other.
>
> Is there some known feature of the releases that would have caused these
> differences? I tried to look over the update notes, but I didn't find
> anything sinister.
>
>
> Thanks,
> Zach
>
> ** Affects: mg5amcnlo
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1731086
>
> Title:
> Change in production xsecs in SUSY models in 2.6.0
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Hi there,
>
> I'm hoping that you can help us track this issue down, as it's holding
> up our validation of MadGraph 2.6.0. Emma Kuwertz and I found that
> between older and newer MadGraph releases, the cross section of p p >
> go go j processes changed by 50% or more. I set up and run out of the
> box:
>
> import model mssm
> generate p p > go go
> add ...

[Olivier Mattelaer (olivier-mattelaer)]

here is the patch

[Zachary Marshall (zach-marshall)]

Thanks Olivier! I see that cuts.f has Windows-style line breaks, which means that the patch when just copied from the web fails. You might want to clean those up in the future.

We're testing the patch out now and will come back soon with news...

Best,
Zach

[Emma Kuwertz (ekuwertz)]

Hi Olivier,

I put this patch into MG 2.6.0 and ran the same standalone test described by Zach above. I get the same (higher) cross-section as before:

Cross-section : 0.7888 +- 0.002189 pb

should I?

Thanks,
Emma

[Emma Kuwertz (ekuwertz)]

Hi Olivier,

any news on this?

Thanks,
Emma

[Olivier Mattelaer (olivier-mattelaer)]

Hi Emma,

Did you generate such file with use_syst=T?
If you do, what is the pdf dependencies for a given event?
(if you copy/paste such information here for one event, I can figure it myself)

To be sure, you are running with pdf reweighting in MLM set on ON right?
(my patch was fixing the fact that such option was most of the time set on OFF even if set on ON in the run_card)

Cheers,

Olivier

> On Nov 22, 2017, at 00:42, Emma Kuwertz <email address hidden> wrote:
>
> Hi Olivier,
>
> any news on this?
>
> Thanks,
> Emma
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1731086
>
> Title:
> Change in production xsecs in SUSY models in 2.6.0
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Hi there,
>
> I'm hoping that you can help us track this issue down, as it's holding
> up our validation of MadGraph 2.6.0. Emma Kuwertz and I found that
> between older and newer MadGraph releases, the cross section of p p >
> go go j processes changed by 50% or more. I set up and run out of the
> box:
>
> import model mssm
> generate p p > go go
> add process p p > go go j
> output -f
>
> in MadGraph 2.3.3 and 2.4.3. I run the same in MadGraph 2.6.0, but
> with `import model MSSM_SLHA2`. In both releases I then set the
> masses of all SUSY particles to be 10**9 GeV, except the gluino which
> is set to 1 TeV. All other parameters I leave as they are, out of the
> box, but a quick check suggests that there aren't wild new cuts that
> have appeared. With 2.3.3 the cross section I get is:
>
> Cross-section : 0.662 +- 0.00212 pb
>
> With 2.4.3 and 2.6.0, I get the same answer:
>
> Cross-section : 0.7888 +- 0.002181 pb
>
> There, the cross section files suggest that most of the difference is
> due to a large (>25%) change in g g > go go g. I don't see any
> difference in the PDFs that might cause such a change, though.
>
> In Emma's run inside the ATLAS framework, we were looking at more
> modest total cross section changes, but large changes to the go go j
> cross section (of order 50%) that led to significantly harder
> kinematics in one case than the other.
>
> Is there some known feature of the releases that would have caused
> these differences? I tried to look over the update notes, but I
> didn't find anything sinister.
>
>
> Thanks,
> Zach
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mg5amcnlo/+bug/1731086/+subscriptions

[Emma Kuwertz (ekuwertz)]

Hi Oliver,

Standalone, for the process Zach described above, I'm now able to reproduce the cross-section we got in 2.3.3 using 2.6.0 with your patch (with the default run_card that has MLM on and use_syst=T).

Unfortunately we still get the same slightly weird results for the individual processes in another job (which has ickkw=0). The difference in p p > go go j cross-sections mentioned in Zach's first message on this thread appears to lead to a difference in kinematics in the final state and we want to be sure that 2.6.0 is doing the right thing.

I made some quick plots - you can see the difference in the jet multiplicity distributions here (both before and after applying your patch to MG 2.6.0):
http://ekuwertz.web.cern.ch/ekuwertz/njets_beforepatch.eps
http://ekuwertz.web.cern.ch/ekuwertz/njets_afterpatch.eps

As you can see, it looks as though the patch does improve the agreement in the tail of the distribution, but it still looks as though there is a systematic effect.

I also put process directories from the 2.3.3 and 2.6.0 jobs here in case you are able to identify anything strange in our setup:
http://ekuwertz.web.cern.ch/ekuwertz/MG-2.6.0.tar.gz
http://ekuwertz.web.cern.ch/ekuwertz/MG-2.3.3.tar.gz

Do you think you could take a look to see whether there is any smoking gun in our setup?

Thanks,
Emma

[Olivier Mattelaer (olivier-mattelaer)]

Hi Emma,

I’m a bit confused about your last message.

> Standalone, for the process Zach described above, I'm now able to
> reproduce the cross-section we got in 2.3.3 using 2.6.0 with your patch
> (with the default run_card that has MLM on and use_syst=T).

Ok that’s a good news.

> I made some quick plots - you can see the difference in the jet multiplicity distributions here (both before and after applying your patch to MG 2.6.0):
> http://ekuwertz.web.cern.ch/ekuwertz/njets_beforepatch.eps
> http://ekuwertz.web.cern.ch/ekuwertz/njets_afterpatch.eps

So those plot are for the case with ickkw=1/MLM merging right?
Looking at Njet distribution is not the most sensitive observables to look at difference between MG version since that observables is dominated by shower effect.
Are you using the same setup for pythia8 in both case? Note that the difference between before and after patch is within theoretical uncertainty.
This shows that such uncertainty is not that small so I would not worry too much about the remaining difference.
Which choice did you take for the renormalization/refactorization scale? If you use the default of MG5, the associate algorithms have been improved from 2.3.3 to 2.6.0.
Making that using that scale choice not suitable for comparison between version.

> I also put process directories from the 2.3.3 and 2.6.0 jobs here in case you are able to identify anything strange in our setup:
> http://ekuwertz.web.cern.ch/ekuwertz/MG-2.6.0.tar.gz
> http://ekuwertz.web.cern.ch/ekuwertz/MG-2.3.3.tar.gz

Those samples are generated with ickkw=0, is this intended? Here I can see that you use the default dynamical_scale_choice meaning that some difference are expected due to that point only.
(within theoretical error obviously)

Actually one good plot to see the change in the scale, would be to plot that scale.

Cheers,

Olivier

> On Nov 23, 2017, at 00:09, Emma Kuwertz <email address hidden> wrote:
>
> Hi Oliver,
>
> Standalone, for the process Zach described above, I'm now able to
> reproduce the cross-section we got in 2.3.3 using 2.6.0 with your patch
> (with the default run_card that has MLM on and use_syst=T).
>
> Unfortunately we still get the same slightly weird results for the
> individual processes in another job (which has ickkw=0). The difference
> in p p > go go j cross-sections mentioned in Zach's first message on
> this thread appears to lead to a difference in kinematics in the final
> state and we want to be sure that 2.6.0 is doing the right thing.
>
> I made some quick plots - you can see the difference in the jet multiplicity distributions here (both before and after applying your patch to MG 2.6.0):
> http://ekuwertz.web.cern.ch/ekuwertz/njets_beforepatch.eps
> http://ekuwertz.web.cern.ch/ekuwertz/njets_afterpatch.eps
>
> As you can see, it looks as though the patch does improve the agreement
> in the tail of the distribution, but it still looks as though there is a
> systematic effect.
>
> I also put process directories from the 2.3.3 and 2.6.0 jobs here in case you are able to identify anything strange in our setup:
> http://ekuwertz.web.ce...

[Emma Kuwertz (ekuwertz)]

Hi Olivier,

thanks a lot for taking a look and apologies for not making it more clear what is shown in the plots I pointed you to. The MG 2.3.3 version of these plots were made using the configuration you see in the MG 2.3.3 process directory, so that’s ickkw=0 and CKKW-L merging (this is what we use by default for our SUSY simplified models in ATLAS). This is compared with MG 2.6.0 with the same thing (ickkw=0), and the process directory for MG 2.6.0 I pointed you to has these same settings and also has your patch applied.

The same pythia version (8.226) is used in all cases. As you pointed out already, we use the default dynamical_scale_choice, so it sounds as though this could possibly explain the remaining differences we see, is that right? Do you have a pointer to some documentation somewhere on the improvements between 2.3.3 and 2.6.0 that you mentioned?

Thanks again,
Emma

[Zachary Marshall (zach-marshall)]

Hi Olivier,

Sorry to be so persistent, but we have a few users breathing down our necks to make sure this issue is resolved. Do you have any documentation for those scale improvements? Anything else you see that's particularly fishy in the setup, or did Emma's answers all make sense?

Best,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

For CKKWL, I’m not sure that there is a problem at all actually (at least not in MG).

I have do the following:
1) mv models/MSSM_SLHA2 models/mssm
2) run the following script (with different version):
 import model mssm
 generate p p > go go
 add process p p > go go j
 output -f
 launch
 set ickkw 0
 set xqcut 0
 set mgo 1000

If i run with version 2.6.1 (not released so after the patch) , I have
  === Results Summary for run: run_01 tag: tag_1 ===

     Cross-section : 0.4667 +- 0.001416 pb
     Nb of events : 10000

INFO: Running Systematics computation
INFO: Idle: 0, Running: 4, Completed: 0 [ current time: 21h31 ]
INFO: Idle: 0, Running: 3, Completed: 1 [ 1m 42s ]
INFO: # events generated with PDF: NNPDF23_lo_as_0130_qed (247000)
INFO: #Will Compute 235 weights per event.
INFO: #***************************************************************************
#
# original cross-section: 0.46658878736
# scale variation: +51.1% -31.7%
# emission scale variation: + 0% -31.7%
# central scheme variation: + 0% -37.1%
# PDF variation: +18.5% -18.5%
#
# dynamical scheme # 1 : 0.395043 +49.4% -30.9% # \sum ET
# dynamical scheme # 2 : 0.313735 +46.1% -29.6% # \sum\sqrt{m^2+pt^2}
# dynamical scheme # 3 : 0.458261 +50.8% -31.5% # 0.5 \sum\sqrt{m^2+pt^2}
# dynamical scheme # 4 : 0.293286 +45.1% -29.2% # \sqrt{\hat s}
#***************************************************************************

If i run with version 2.3.3, I have:
  === Results Summary for run: run_01 tag: tag_1 ===

     Cross-section : 0.4671 +- 0.001455 pb
     Nb of events : 10000

If i run with version 2.4.1, I have:
  === Results Summary for run: run_01 tag: tag_1 ===

     Cross-section : 0.4667 +- 0.001416 pb
     Nb of events : 10000

If i run with version 2.6.0:
  === Results Summary for run: run_01 tag: tag_1 ===

     Cross-section : 0.4667 +- 0.001416 pb
     Nb of events : 10000
INFO: #***************************************************************************
#
# original cross-section: 0.46658878736
# scale variation: +51.2% -31.7%
# emission scale variation: + 0% -31.7%
# central scheme variation: + 0% -37.2%
# PDF variation: +18.5% -18.5%
#
# dynamical scheme # 1 : 0.394494 +49.5% -31% # \sum ET
# dynamical scheme # 2 : 0.3136 +46.1% -29.6% # \sum\sqrt{m^2+pt^2}
# dynamical scheme # 3 : 0.4583 +50.9% -31.6% # 0.5 \sum\sqrt{m^2+pt^2}
# dynamical scheme # 4 : 0.293178 +45.2% -29.2% # \sqrt{\hat s}
#***************************************************************************

So here I do not see anything different.
As you can see (and as I was expecting my patch has not effect at all in the case of ickkw=0)
Since my patch was related to the pdf reweighting of the MLM matching (ickkw=1).

I also check the result for each type of final state:
here it is for 2.6.0:

Graph Cross-Section ↓ Error Events (K) Unwgt Luminosity
P2 sum 0.29763812117
P2_gg_gogog 0.2134 0.00117 116.599 8689.0 0
P1 sum 0.169090624
P1_gg_gogo 0.1469 0.000425 89.093 5333.0 0
P2_gq_gogoq 0.06981 0.000667 315.828 24045.0 0
P1_qq_gogo 0.02222 0.000109 63.122 3782.0...

[Emma Kuwertz (ekuwertz)]

Dear Olivier,

thanks for your reply and for your checks. I tried to replicate your test in 2.6.0 and 2.3.3, but this time included a cut on ktdurham (as is used in our setup).

I ran the following:
 import model mssm
 generate p p > go go
 add process p p > go go j
 output -f
 launch
 set ickkw 0
 set xqcut 0
 set mgo 1000
 set ktdurham 250

It looks as though you have been able to set ktdurham since 2.0.0, which we understood is needed for CKKW-L merging?

The results we get are as follows:

2.6.0
cross-section 0.2005 pm 0.000402 pb

2.3.3
cross-section 0.19147 pm 0.000345 pb

Here the p2 cross-sections increase by 50% in 2.3.3 wrt 2.6.0.

We didn't make any hack to 2.3.3 in order to deal with CKKW-L merging, we understood that the configuration should work as is. I guess the question is did the behaviour of the ktdurham variable change significantly between 2.3.3 and 2.6.0?

Thanks,
Emma & Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

I actually learned something today, indeed this cut is present in 2.3.3.
I will discuss with Valentin Hirschi and Stepan Pretzel to see what change they did to that cut when they
add the official support of CKKW-L in 2.5.0.

Cheers,

Olivier

> On Nov 30, 2017, at 13:15, Emma Kuwertz <email address hidden> wrote:
>
> Dear Olivier,
>
> thanks for your reply and for your checks. I tried to replicate your
> test in 2.6.0 and 2.3.3, but this time included a cut on ktdurham (as is
> used in our setup).
>
> I ran the following:
> import model mssm
> generate p p > go go
> add process p p > go go j
> output -f
> launch
> set ickkw 0
> set xqcut 0
> set mgo 1000
> set ktdurham 250
>
> It looks as though you have been able to set ktdurham since 2.0.0, which
> we understood is needed for CKKW-L merging?
>
> The results we get are as follows:
>
> 2.6.0
> cross-section 0.2005 pm 0.000402 pb
>
> 2.3.3
> cross-section 0.19147 pm 0.000345 pb
>
> Here the p2 cross-sections increase by 50% in 2.3.3 wrt 2.6.0.
>
> We didn't make any hack to 2.3.3 in order to deal with CKKW-L merging,
> we understood that the configuration should work as is. I guess the
> question is did the behaviour of the ktdurham variable change
> significantly between 2.3.3 and 2.6.0?
>
> Thanks,
> Emma & Zach
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1731086
>
> Title:
> Change in production xsecs in SUSY models in 2.6.0
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Hi there,
>
> I'm hoping that you can help us track this issue down, as it's holding
> up our validation of MadGraph 2.6.0. Emma Kuwertz and I found that
> between older and newer MadGraph releases, the cross section of p p >
> go go j processes changed by 50% or more. I set up and run out of the
> box:
>
> import model mssm
> generate p p > go go
> add process p p > go go j
> output -f
>
> in MadGraph 2.3.3 and 2.4.3. I run the same in MadGraph 2.6.0, but
> with `import model MSSM_SLHA2`. In both releases I then set the
> masses of all SUSY particles to be 10**9 GeV, except the gluino which
> is set to 1 TeV. All other parameters I leave as they are, out of the
> box, but a quick check suggests that there aren't wild new cuts that
> have appeared. With 2.3.3 the cross section I get is:
>
> Cross-section : 0.662 +- 0.00212 pb
>
> With 2.4.3 and 2.6.0, I get the same answer:
>
> Cross-section : 0.7888 +- 0.002181 pb
>
> There, the cross section files suggest that most of the difference is
> due to a large (>25%) change in g g > go go g. I don't see any
> difference in the PDFs that might cause such a change, though.
>
> In Emma's run inside the ATLAS framework, we were looking at more
> modest total cross section changes, but large changes to the go go j
> cross section (of order 50%) that led to significantly harder
> kinematics in one case than the other.
>
> Is there some known feature of the releases that would have caused
> these differences? I tried to look over the update notes, but I
> didn't find anything sinister.
>
>
> Than...

[Emma Kuwertz (ekuwertz)]

Hi Olivier,

we were wondering whether you'd had any feedback on this yet? We need to launch things on our side very soon, so if you don’t anticipate a resolution in the next couple of days we might have to revert to MadGraph 2.4.3.

Cheers,
Emma & Zach