Reproducibility problem in 2.6.2 (in combine?)

[Zachary Marshall]

Dear authors,

I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/

We see some non-reproducibility, even when setting a random number seed. I attach a tarball of the cards directories so that you have the run, param, and proc cards. For the actual runs, iseed was set to 1234 (of course, the code resets it to 0 after the run).

Looking at the LHE files, they seem to contain the same events, but in different order. Just diffing the two files, you can see an example right away:

1487a1488,1496
> 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
> 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
> 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
> 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
> 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
> 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
> 1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
> </event>
> <event>
1506,1532c1515,1542
< 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
< 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
< 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
< 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
< 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
< 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
< 1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
< </event>

I believe that's the same event, but note that the line numbers are ~20 apart -- this event is first in one file and fourth in the other. I believe this indicates some non-reproducibility or a race condition in combine somewhere.

Because these events then enter programs like Pythia8 -- or even when they enter MadSpin -- the events are processed in order, the final events then appear random (different seeds are applied to the same particles). That means the events are effectively not reproducible when processed through, so this is an issue for our bug hunting and for various other reasons.

If you could help us out with this it would be very much appreciated!

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

This is actually the case since a couple of version.
If you want to ensure 100% reproducibility you also have to set the python seed in top of the fortran seed that are set within the run_card.

Cheers,

Olivier

> On 4 May 2018, at 23:23, Zachary Marshall <email address hidden> wrote:
>
> Public bug reported:
>
> Dear authors,
>
> I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
> /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/
>
> We see some non-reproducibility, even when setting a random number seed.
> I attach a tarball of the cards directories so that you have the run,
> param, and proc cards. For the actual runs, iseed was set to 1234 (of
> course, the code resets it to 0 after the run).
>
> Looking at the LHE files, they seem to contain the same events, but in
> different order. Just diffing the two files, you can see an example
> right away:
>
> 1487a1488,1496
>> 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
>> 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
>> 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
>> 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
>> 1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> </event>
>> <event>
> 1506,1532c1515,1542
> < 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
> < 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
> < 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
> < 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
> < 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
> < 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
> < 1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
> < </event>
>
>
> I believe that's the same event, but note that the line numbers are ~20 apart -- this event is first in one file and fourth in the other. I believe this indicates some non-reproducibility or a race condition in combine somewhere.
>
> Becau...

[Zachary Marshall (zach-marshall)]

Thanks Olivier, but I don't understand your answer. At the top of the run card is:

#*********************************************************************
# Number of events and rnd seed *
# Warning: Do not generate more than 1M events in a single run *
# If you want to run Pythia, avoid more than 50k events in a run. *
#*********************************************************************
  24000 = nevents ! Number of unweighted events requested
  1234 = iseed ! rnd seed (0=assigned automatically=default))

I only see a single seed here, and it's unclear where the distinction is happening that you seem to indicate between a python seed and a fortran seed.

Best,
Zach

[Zachary Marshall (zach-marshall)]

Just a quick nudge. I looked through the code and don't see any indication of another seed in the run card. I looked through the update notes and as much documentation as I could find and I don't see any indication of either a different seed or a change in behavior. I looked through the uses of random, and most seem to seed off of iseed (as I'd hoped). I hope you don't mean that we now need to hack the code to add a setting of the seed for the python random module based on iseed -- surely that should be centrally done.

Thanks again,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

That's what I mean. If you want full reproductibility you can add the following line:

=== modified file 'madgraph/interface/common_run_interface.py'
--- madgraph/interface/common_run_interface.py 2018-04-28 21:38:38 +0000
+++ madgraph/interface/common_run_interface.py 2018-05-07 18:58:48 +0000
@@ -5607,6 +5607,8 @@
         """This is run on quitting the class. Apply here all the self-consistency
         rule that you want. Do the modification via the set command."""

+ import random
+ random.seed(self.run_card['iseed'])
         ########################################################################
         # LO specific check
         ########################################################################

You can actually set such random see basically anywhere (especially if you always set it to zero which is fine as well). In that case, you can set it to zero within ./bin/mg5_aMC

Cheers,

Olivier

> On 7 May 2018, at 20:40, Zachary Marshall <email address hidden> wrote:
>
> Just a quick nudge. I looked through the code and don't see any
> indication of another seed in the run card. I looked through the update
> notes and as much documentation as I could find and I don't see any
> indication of either a different seed or a change in behavior. I looked
> through the uses of random, and most seem to seed off of iseed (as I'd
> hoped). I hope you don't mean that we now need to hack the code to add
> a setting of the seed for the python random module based on iseed --
> surely that should be centrally done.
>
> Thanks again,
> Zach
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1769271
>
> Title:
> Reproducibility problem in 2.6.2 (in combine?)
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Dear authors,
>
> I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
> /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/
>
> We see some non-reproducibility, even when setting a random number
> seed. I attach a tarball of the cards directories so that you have
> the run, param, and proc cards. For the actual runs, iseed was set to
> 1234 (of course, the code resets it to 0 after the run).
>
> Looking at the LHE files, they seem to contain the same events, but in
> different order. Just diffing the two files, you can see an example
> right away:
>
> 1487a1488,1496
>> 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
>> 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
>> 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
>> 10000...

[Josh McFayden (mcfayden)]

Hi Olivier,

Maybe we've not made it very clear before, but we always want full reproducibility based on the random seed. What reason is there not to have it?

Best,

Josh

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

My main motivation is to further reduce potential bias due to random number.
Due to the strength of the random generator it actually improves the strength of the method
rather than reducing it. This was introduced a while ago (something like 2 year ago) actually in normal generation mode but was introduced into gridpack only more recently (due to some request of CMS of doing
standalone gridpack).

You should not be surprised since we spoke about it (at least two times) in the past.

If adding
import random
random.seed(0)
in the executable or in the python script calling MG5aMC (if you do not call it from shell)
is a problem for you, I can add an hidden switch in the run_card for version 2.6.3 (actually I might do it anyway).

Cheers,

Olivier

> On 7 May 2018, at 21:54, Josh McFayden <email address hidden> wrote:
>
> Hi Olivier,
>
> Maybe we've not made it very clear before, but we always want full
> reproducibility based on the random seed. What reason is there not to
> have it?
>
> Best,
>
> Josh
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1769271
>
> Title:
> Reproducibility problem in 2.6.2 (in combine?)
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Dear authors,
>
> I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
> /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/
>
> We see some non-reproducibility, even when setting a random number
> seed. I attach a tarball of the cards directories so that you have
> the run, param, and proc cards. For the actual runs, iseed was set to
> 1234 (of course, the code resets it to 0 after the run).
>
> Looking at the LHE files, they seem to contain the same events, but in
> different order. Just diffing the two files, you can see an example
> right away:
>
> 1487a1488,1496
>> 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
>> 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
>> 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
>> 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
>> 1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
>> </event>
>> <event>
> 1506,1532c1515,1542
> < 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
> < 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
> < 2 -1 0...

[Zachary Marshall (zach-marshall)]

Hi Oliver,

Yes, we're usually calling things from a shell. I would think this would also make life easier on you all -- ensuring good reproducibility means that you can more easily debug problems when they arise. Having a switch in the run card would be fine with me.

Best,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

The (hidden) additional parameter is coded in the following push/patch:
https://bazaar.launchpad.net/~mg5core2/mg5amcnlo/2.6.3/revision/279

Cheers,

Olivier

On 7 May 2018, at 22:44, Zachary Marshall <<email address hidden><mailto:<email address hidden>>> wrote:

Hi Oliver,

Yes, we're usually calling things from a shell. I would think this
would also make life easier on you all -- ensuring good reproducibility
means that you can more easily debug problems when they arise. Having a
switch in the run card would be fine with me.

Best,
Zach

--
You received this bug notification because you are subscribed to
MadGraph5_aMC@NLO.
https://bugs.launchpad.net/bugs/1769271

Title:
 Reproducibility problem in 2.6.2 (in combine?)

Status in MadGraph5_aMC@NLO:
 New

Bug description:
 Dear authors,

 I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
 /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/

 We see some non-reproducibility, even when setting a random number
 seed. I attach a tarball of the cards directories so that you have
 the run, param, and proc cards. For the actual runs, iseed was set to
 1234 (of course, the code resets it to 0 after the run).

 Looking at the LHE files, they seem to contain the same events, but in
 different order. Just diffing the two files, you can see an example
 right away:

 1487a1488,1496
6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
      21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
       2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
      24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
       1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
</event>
<event>
 1506,1532c1515,1542
 < 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
 < 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
 < 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
 < 24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
 < 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
 < 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000...

[Hannes (hannes3)]

Hi,

I am setting both seed and python_seed in the run_card.dat in MG5 2.6.5 but the events are still shuffled, is there something I am missing? I have attached a script to reproduce the issue.

Best wishes,
Hannes

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Ok this was a very difficult issue.
So the code was actually deterministic but it requires additional condition to produce the same sample
1) running on the same computer
2) running on the same path

The randomness is linked to the actual randomness of python which allow code to find element at order 1 in a set
independently of the size of the set. So requesting reproducibility enforces us to slow down the code.
(the loss will likely be negligible but I do not like that)

You will find the patch here:
https://bazaar.launchpad.net/~mg5core1/mg5amcnlo/2.6.6/revision/299

Cheers,

Olivier

On 27 Mar 2019, at 15:18, Hannes <<email address hidden><mailto:<email address hidden>>> wrote:

Hi,

I am setting both seed and python_seed in the run_card.dat in MG5 2.6.5
but the events are still shuffled, is there something I am missing? I
have attached a script to reproduce the issue.

Best wishes,
Hannes

** Attachment added: "run.sh"
  https://bugs.launchpad.net/mg5amcnlo/+bug/1769271/+attachment/5249826/+files/run.sh

--
You received this bug notification because you are subscribed to
MadGraph5_aMC@NLO.
https://bugs.launchpad.net/bugs/1769271

Title:
 Reproducibility problem in 2.6.2 (in combine?)

Status in MadGraph5_aMC@NLO:
 Fix Released

Bug description:
 Dear authors,

 I'm running tests with MG5_aMC 2.6.2 - in case you have a CERN account, it's this one:
 /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.6.2.atlas/x86_64-slc6-gcc47-opt/

 We see some non-reproducibility, even when setting a random number
 seed. I attach a tarball of the cards directories so that you have
 the run, param, and proc cards. For the actual runs, iseed was set to
 1234 (of course, the code resets it to 0 after the run).

 Looking at the LHE files, they seem to contain the same events, but in
 different order. Just diffing the two files, you can see an example
 right away:

 1487a1488,1496
6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
      21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
       2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.6619322556e+03 1.6619322556e+03 0.0000000000e+00 0.0000e+00 -1.0000e+00
      24 2 1 2 0 0 +8.9075749711e+01 +3.5726986208e+01 -1.0898807218e+03 1.1239759236e+03 2.5743150990e+02 0.0000e+00 0.0000e+00
 1000023 1 3 3 0 0 +8.0116545705e+01 +1.1392922116e+02 -6.6081705506e+02 6.8029530983e+02 8.2000000000e+01 0.0000e+00 -1.0000e+00
 1000024 1 3 3 0 0 +8.9592040060e+00 -7.8202234949e+01 -4.2906366674e+02 4.4368061374e+02 8.1000000000e+01 0.0000e+00 1.0000e+00
       1 1 1 2 501 0 -8.9075749711e+01 -3.5726986208e+01 -5.5085490525e+02 5.5915296062e+02 0.0000000000e+00 0.0000e+00 -1.0000e+00
</event>
<event>
 1506,1532c1515,1542
 < 6 1 +8.9242237e-05 3.75379100e+02 7.95049700e-03 1.05265500e-01
 < 21 -1 0 0 501 502 +0.0000000000e+00 +0.0000000000e+00 +2.1196628570e+01 2.1196628570e+01 0.0000000000e+00 0.0000e+00 -1.0000e+00
 < 2 -1 0 0 502 0 -0.0000000000e+00 -0.0000000000e+00 -1.66193225...