Madspin doesn't save and load pickle file properly for recent MG5v3

[Sitian Qian]

When we use the recent MG5v33x versions, we find Madspin cannot run properly from a pickle file. The error reads

Traceback (most recent call last):
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/bin/test_ttbar/Events/run_01/../../../../MadSpin/madspin", line 139, in <module>
    cmd_line.import_command_file(input_file)
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1692, in import_command_file
    self.exec_cmd(line, precmd=True)
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1573, in exec_cmd
    stop = Cmd.onecmd_orig(current_interface, line, **opt)
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1493, in onecmd_orig
    return func(arg, **opt)
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/various/misc.py", line 110, in f_with_no_logger
    out = f(self, *args, **opt)
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/MadSpin/interface_madspin.py", line 621, in do_launch
    return self.run_from_pickle()
  File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/MadSpin/interface_madspin.py", line 718, in run_from_pickle
    d['decay_struct'] = eval(d['decay_struct'])
TypeError: eval() arg 1 must be a string, bytes or code object

However, it runs successfully with on-the-fly usages. After looking into the codes, we find out that this bug comes from the improper treatment for pickle save and load part. In which the 'decay_struct' dictionaries are converted to strings when saving pickle file and convert back when loading. This leads to the bug mentioned above.

Now after commenting out the corresponding lines MadSpin works properly with saved pickle files. Changes are listed below:

--- a/MadSpin/decay.py
+++ b/MadSpin/decay.py
@@ -2181,9 +2181,9 @@ class decay_all_events(object):
         #self.all_ME, bkp2 = None , self.all_ME
         #self.width_estimator, bkp3 = self.width_estimator, self.width_estimator
         modelpath = model.get('modelpath')
- for me in self.all_ME:
- for d in self.all_ME[me]['decays']:
- d['decay_struct'] = str(d['decay_struct'])
+ #for me in self.all_ME:
+ # for d in self.all_ME[me]['decays']:
+ # d['decay_struct'] = str(d['decay_struct'])
         self.modelpath = modelpath

--- a/MadSpin/interface_madspin.py
+++ b/MadSpin/interface_madspin.py
@@ -713,9 +713,9 @@ class MadSpinInterface(extended_cmd.Cmd):

         #restore data passed to string to help pickle
         generate_all.all_decay = eval(generate_all.all_decay)
- for me in generate_all.all_ME:
- for d in generate_all.all_ME[me]['decays']:
- d['decay_struct'] = eval(d['decay_struct'])
+ #for me in generate_all.all_ME:
+ # for d in generate_all.all_ME[me]['decays']:
+ # d['decay_struct'] = eval(d['decay_struct'])

         # Re-create information which are not save in the pickle.

[Olivier Mattelaer (olivier-mattelaer)]

Then this depend of the madspin_card that you use, since the linearisation of that dictionary can sometimes lead to crash and that was the reason of such patch.

The real question is why "decay_struct" is not a string when set like one...
Are you using the spinmode only mode of madspin?

Cheers,

Olivier

> On 23 Dec 2021, at 08:04, Sitian Qian <email address hidden> wrote:
>
> Public bug reported:
>
> When we use the recent MG5v33x versions, we find Madspin cannot run
> properly from a pickle file. The error reads
>
> Traceback (most recent call last):
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/bin/test_ttbar/Events/run_01/../../../../MadSpin/madspin", line 139, in <module>
> cmd_line.import_command_file(input_file)
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1692, in import_command_file
> self.exec_cmd(line, precmd=True)
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1573, in exec_cmd
> stop = Cmd.onecmd_orig(current_interface, line, **opt)
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/interface/extended_cmd.py", line 1493, in onecmd_orig
> return func(arg, **opt)
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/madgraph/various/misc.py", line 110, in f_with_no_logger
> out = f(self, *args, **opt)
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/MadSpin/interface_madspin.py", line 621, in do_launch
> return self.run_from_pickle()
> File "/data/bond/stqian/reproduce_MG5331/MG5_aMC_v3_3_1/MadSpin/interface_madspin.py", line 718, in run_from_pickle
> d['decay_struct'] = eval(d['decay_struct'])
> TypeError: eval() arg 1 must be a string, bytes or code object
>
> However, it runs successfully with on-the-fly usages. After looking into
> the codes, we find out that this bug comes from the improper treatment
> for pickle save and load part. In which the 'decay_struct' dictionaries
> are converted to strings when saving pickle file and convert back when
> loading. This leads to the bug mentioned above.
>
> Now after commenting out the corresponding lines MadSpin works properly
> with saved pickle files. Changes are listed below:
>
> --- a/MadSpin/decay.py
> +++ b/MadSpin/decay.py
> @@ -2181,9 +2181,9 @@ class decay_all_events(object):
> #self.all_ME, bkp2 = None , self.all_ME
> #self.width_estimator, bkp3 = self.width_estimator, self.width_estimator
> modelpath = model.get('modelpath')
> - for me in self.all_ME:
> - for d in self.all_ME[me]['decays']:
> - d['decay_struct'] = str(d['decay_struct'])
> + #for me in self.all_ME:
> + # for d in self.all_ME[me]['decays']:
> + # d['decay_struct'] = str(d['decay_struct'])
> self.modelpath = modelpath
>
>
> --- a/MadSpin/interface_madspin.py
> +++ b/MadSpin/interface_madspin.py
> @@ -713,9 +713,9 @@ class MadSpinInterface(extended_cmd.Cmd):
>
> #restore data passed to string to help pickle
> generate_all.all_decay = eval(generate_all.all_decay)
> - for me in generate_all.all_ME:
> -...

[Sitian Qian (sitianq)]

Thanks for your quick response!

In fact we saw this issue firstly with dummy decay mode. Then we found this also happens when decaying on-shell particles. For example the errors I posted is produced for decaying tt~ events with the default settings.

I am not confident to answer if I am using spinmode only, but if the default setting is then yes I am using it, and also I am using dummy decay mode to modify the helicity.

I am relooking at the files and I feel the problem is that even if the "decay_struct" are stored after being converted to string. Once they are loaded they are converted back to dictionaries.

I add a try except to the loading part:

 716 for me in generate_all.all_ME:
 717 for d in generate_all.all_ME[me]['decays']:
 718 try:
 719 tmp_ds = eval(d['decay_struct'])
 720 except TypeError:
 721 print("eval error")
 722 if d['decay_struct'] == "":
 723 print("decay_struct is empty string")
 724 else:
 725 print(type(d['decay_struct']))
 726 else:
 727 d['decay_struct'] = tmp_ds

With this try except part MadSpin now works and when I look at the output file I find the 'decay_struct' are all dictionaries.

Cheers,

Sitian

[Olivier Mattelaer (olivier-mattelaer)]

Thanks for the comment,
I have pushed the fix in the development version of the long term stable branch:
https://bazaar.launchpad.net/~maddevelopers/mg5amcnlo/LTS_dev/revision/375

Thanks,

Olivier