Two errors: use_syst and jet matching in multicore run

[Motoi Endo]

Using Python3 versions of madgraph, v2.7.3.py3 and v2.8.3.2, I found the following two independent errors.
The sample script is simple:

------------------
generate p p > t t~
add process p p > t t~ j
output TEST

launch TEST
shower=Pythia8
done
set run_card nevents 500
done
------------------

1. Error in turning off use_syst:

When I added the following in the above madevent run,

set run_card use_syst False

An error is raised:

MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4068
in setup_Pythia8RunAndCard

    if scale<(1.5*self.run_card['xqcut']):

TypeError: '<' not supported between instances of 'str' and 'float'

The reason is the following:

MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4064-

            for scale in PY8_Card['SysCalc:qCutList']:
                if scale<(1.5*self.run_card['xqcut']):
                    logger.error(
        'One of the MLM merging qCut parameter you chose (%f) in the variation list'%scale+¥
        " (either via 'SysCalc:qCutList' in the PY8 shower card or "+¥
        "'sys_matchscale' in the run_card) is less than 1.5*xqcut, where xqcut is"+
        ' the run_card parameter (=%f)¥n'%self.run_card['xqcut']+
        'It would be better/safer to use a larger qCut or a smaller xqcut.')

When self.run_card['use_syst'] is False and PY8_Card['SysCalc:qCutList'] is 'auto' (default setting), the line 4065 ('if scale<(1.5*...') raises an error because the parameter 'scale' in the line 4064 becomes 'auto', whose type is 'str', and compared with a 'float' type object. (This error is not raised in Python2 because 'str' is allowed to be compared with 'float'.)

2. Error in calculating merged cross sections at jet matching with multicore run

When I tried to perform a multicore run for the jet matching, for instance at N core runs, the final merged cross section becomes N times larger than the result of the single core run. I checked this happening in Mac and Ubuntu, and found out the reason; the Pythia card parameters are stored doubly in 'ParallelPY8Card' after 'copy.copy' at madevent_interface.py: line 4439. This problem originates in 'ConfigFile' in banner.py:

MG5_aMC_v2_8_3_2/madgraph/various/banner.py: line 968-

class ConfigFile(dict):
    """ a class for storing/dealing with input file.
    """

    def __init__(self, finput=None, **opt):
        """initialize a new instance. input can be an instance of MadLoopParam,
        a file, a path to a file, or simply Nothing"""

        if isinstance(finput, self.__class__):
            dict.__init__(self, finput)
            for key in finput.__dict__:
                setattr(self, key, copy.copy(getattr(finput, key)) )
            for key,value in finput.items():
                dict.__setitem__(self, key.lower(), value)
            return
        else:
            dict.__init__(self)

When ConfigFile is copied by copy.copy, for example, ParallelPY8Card = copy.copy(PY8_Card), the function, def __init__(self, finput=None, **opt), is called with 'finput' specified. Here, the keys of 'finput' (via finput.items()) are composed by upper- and lower-case characters such as 'Main:numberOfEvents'. Then, the keys of 'self' after the line, dict.__init__(self, finput), include the uppercase characters. Consequently, all the keys including the uppercase characters are duplicated after the line, dict.__setitem__(self, key.lower(), value), because they are set in lowercase. This causes problems, for instance, when one tries to perform multicore Pythia run, because 'Main:numberOfEvents' and 'HEPMCoutput:scaling' are not set appropriately. The duplication may be resolved by modifying the code as

            dict.__init__(self, finput)
            for key in finput.__dict__:
                setattr(self, key, copy.copy(getattr(finput, key)) )
            for key,value in finput.items():
                dict.__delitem__(self, key)
                dict.__setitem__(self, key.lower(), value)
            return

where I added dict.__delitem__(self, key). Then, the pythia card for the parallel runs looks to be fixed, and the merged cross sections become consistent with those in the single core run.

I hope these two reports would be helpful for you.

Best,
Motoi

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Thanks a lot for those bug report and investigation on those, this is very helpful.

I was not aware of the first one and will investigate it.
The second bug was actually already identified and already fixed in the latest version of the code.

Thanks,

Olivier

> On 27 Feb 2021, at 08:27, Motoi Endo <email address hidden> wrote:
>
> Public bug reported:
>
> Using Python3 versions of madgraph, v2.7.3.py3 and v2.8.3.2, I found the following two independent errors.
> The sample script is simple:
>
> ------------------
> generate p p > t t~
> add process p p > t t~ j
> output TEST
>
> launch TEST
> shower=Pythia8
> done
> set run_card nevents 500
> done
> ------------------
>
> 1. Error in turning off use_syst:
>
> When I added the following in the above madevent run,
>
> set run_card use_syst False
>
> An error is raised:
>
> MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4068
> in setup_Pythia8RunAndCard
>
> if scale<(1.5*self.run_card['xqcut']):
>
> TypeError: '<' not supported between instances of 'str' and 'float'
>
> The reason is the following:
>
> MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4064-
>
> for scale in PY8_Card['SysCalc:qCutList']:
> if scale<(1.5*self.run_card['xqcut']):
> logger.error(
> 'One of the MLM merging qCut parameter you chose (%f) in the variation list'%scale+¥
> " (either via 'SysCalc:qCutList' in the PY8 shower card or "+¥
> "'sys_matchscale' in the run_card) is less than 1.5*xqcut, where xqcut is"+
> ' the run_card parameter (=%f)¥n'%self.run_card['xqcut']+
> 'It would be better/safer to use a larger qCut or a smaller xqcut.')
>
> When self.run_card['use_syst'] is False and PY8_Card['SysCalc:qCutList']
> is 'auto' (default setting), the line 4065 ('if scale<(1.5*...') raises
> an error because the parameter 'scale' in the line 4064 becomes 'auto',
> whose type is 'str', and compared with a 'float' type object. (This
> error is not raised in Python2 because 'str' is allowed to be compared
> with 'float'.)
>
> 2. Error in calculating merged cross sections at jet matching with
> multicore run
>
> When I tried to perform a multicore run for the jet matching, for
> instance at N core runs, the final merged cross section becomes N times
> larger than the result of the single core run. I checked this happening
> in Mac and Ubuntu, and found out the reason; the Pythia card parameters
> are stored doubly in 'ParallelPY8Card' after 'copy.copy' at
> madevent_interface.py: line 4439. This problem originates in
> 'ConfigFile' in banner.py:
>
> MG5_aMC_v2_8_3_2/madgraph/various/banner.py: line 968-
>
> class ConfigFile(dict):
> """ a class for storing/dealing with input file.
> """
>
> def __init__(self, finput=None, **opt):
> """initialize a new instance. input can be an instance of MadLoopParam,
> a file, a path to a file, or simply Nothing"""
>
> if isinstance(finput, self.__class__):
> dict.__init__(self, finput)
> for key in finput.__dict__:
> setattr(self, key, copy.copy(getattr(finp...

[Motoi Endo (motoiendo)]

Hi Olivier,

Thank you for the update. I did not aware of it.
I checked v2.9.2 and noticed that copy.copy does not work for the PY8Card class:

MG5_aMC_v2_9_2/madgraph/interface/madevent_interface.py: line 4455

                ParallelPY8Card = copy.copy(PY8_Card)

The reason is following: In the PY8Card class, the object is converted to a different (dict) class at the begging of the copy action. Then, the copy fails in the ConfigFile class because isinstance(finput, self.__class__) returns false:

MG5_aMC_v2_9_2/madgraph/various/banner.py: line 972-

    def __init__(self, finput=None, **opt):
        """initialize a new instance. input can be an instance of MadLoopParam,
        a file, a path to a file, or simply Nothing"""

        if isinstance(finput, self.__class__):
            dict.__init__(self, finput)
            for key in finput.__dict__:
                setattr(self, key, copy.copy(getattr(finput, key)) )
            for key,value in finput.items():
                dict.__setitem__(self, key.lower(), value)
            return
        else:
            dict.__init__(self)

As a result, PY8_Card is not copied to ParallelPY8Card.

Thanks,

Motoi

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Thanks so much for this.

Both issue are fixed in this update (you will have patch/ link to that version/...):
https://bazaar.launchpad.net/~maddevelopers/mg5amcnlo/2.9.3/revision/303 <https://bazaar.launchpad.net/~maddevelopers/mg5amcnlo/2.9.3/revision/303>

Thanks a lot,

Olivier

> On 28 Feb 2021, at 16:42, Motoi Endo <email address hidden> wrote:
>
> Hi Olivier,
>
> Thank you for the update. I did not aware of it.
> I checked v2.9.2 and noticed that copy.copy does not work for the PY8Card class:
>
> MG5_aMC_v2_9_2/madgraph/interface/madevent_interface.py: line 4455
>
> ParallelPY8Card = copy.copy(PY8_Card)
>
> The reason is following: In the PY8Card class, the object is converted
> to a different (dict) class at the begging of the copy action. Then, the
> copy fails in the ConfigFile class because isinstance(finput,
> self.__class__) returns false:
>
> MG5_aMC_v2_9_2/madgraph/various/banner.py: line 972-
>
> def __init__(self, finput=None, **opt):
> """initialize a new instance. input can be an instance of MadLoopParam,
> a file, a path to a file, or simply Nothing"""
>
> if isinstance(finput, self.__class__):
> dict.__init__(self, finput)
> for key in finput.__dict__:
> setattr(self, key, copy.copy(getattr(finput, key)) )
> for key,value in finput.items():
> dict.__setitem__(self, key.lower(), value)
> return
> else:
> dict.__init__(self)
>
> As a result, PY8_Card is not copied to ParallelPY8Card.
>
> Thanks,
>
> Motoi
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1917117
>
> Title:
> Two errors: use_syst and jet matching in multicore run
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Using Python3 versions of madgraph, v2.7.3.py3 and v2.8.3.2, I found the following two independent errors.
> The sample script is simple:
>
> ------------------
> generate p p > t t~
> add process p p > t t~ j
> output TEST
>
> launch TEST
> shower=Pythia8
> done
> set run_card nevents 500
> done
> ------------------
>
> 1. Error in turning off use_syst:
>
> When I added the following in the above madevent run,
>
> set run_card use_syst False
>
> An error is raised:
>
> MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4068
> in setup_Pythia8RunAndCard
>
> if scale<(1.5*self.run_card['xqcut']):
>
> TypeError: '<' not supported between instances of 'str' and 'float'
>
> The reason is the following:
>
> MG5_aMC_v2_8_3_2/madgraph/interface/madevent_interface.py: line 4064-
>
> for scale in PY8_Card['SysCalc:qCutList']:
> if scale<(1.5*self.run_card['xqcut']):
> logger.error(
> 'One of the MLM merging qCut parameter you chose (%f) in the variation list'%scale+¥
> " (either via 'SysCalc:qCutList' in the PY8 shower card or "+¥
> "'sys_matchscale' in the run_card) is less than 1.5*xqcut, where xqcut is"+
> ' the run_card parameter (=%f)¥n'%self.run_card['xq...

[Motoi Endo (motoiendo)]

Hi Olivier,

I investigated the update and the patch works fine. Both the problems are solved.

Thanks,

Motoi