Unphysical event weights not flagged

[Andy Buckley]

Running a Rivet tutorial last week, we hit obstacles with the MG5 event output from your embedded Pythia main program, because the weight vector is non-standard.

In particular, when running with merging enabled, there are three weight streams named `scomp=...`, `smin=...`, and `smax=...` which have weight sums many orders of magnitude larger than the physical weights, and which hence dominate resulting plots to the exclusion of the physics lines.

The weights standard https://arxiv.org/pdf/2203.08230.pdf specifies that unphysical or non-cross-section weight names should be prefixed with `EXTRA_` or `IRREG_` so downstream tools know not to use them in physics plots or stats calculations (see p6 of the linked document). Can MG_aMC be updated to flag these weights for non-plotting with whichever prefix you think fits best?

It would also be nice to have the `Weight` prefix, which is used as the nominal, at the first place in the weight vector as is intended, but we do detect it due to using that agreed name. As far as I can see it's just the `s...` weights that are really to be avoided.

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Is this for some MLM process where you do have use_syst set on True in the run_card?

[Olivier Mattelaer (olivier-mattelaer)]

Looking at those lines it indeed seems coming from pythia8 handling of MLM code (from S. Prestel):
https://github.com/mg5amcnlo/MG5aMC_PY8_interface/blob/a7b38a861a765a720e35311af0c83bb87bbd980a/MG5aMC_PY8_interface.cc#L707

Now the first question is why such flag are included in the hepmc...
They are certainly needed at the lhef level but not sure why Stephan include those in the hepmc.

More important question (but related) is which code are going to break if we change them.
The most likely issue would come from CMSSW. Will contact them to check...

Cheers,

Olivier

[Andy Buckley (andy-insectnation)]

Hi Olivier! I think that Py8 interface code is out of date now. HepMC's weight-ordering issue was fixed some time ago, so it can be correctly structured without being forced into alphabetical ordering: this code seems to be working round an issue that no longer exists. And of course it pre-dates the weight standard.

So the simplest thing would just be to update this interface to set the weights to names like "IRREG_scomp=...". That should be sufficient to get it working, but if the ordering could also be revisited -- now that that's possible. I note this whole thing is inside a "#ifdef HEPMC2HACK" preprocessor block, which is maybe only a workaround for specific HepMC versions, or at least is deactivated for HepMC3, so we should make sure that whatever fix is applied also works when you migrate to running to HepMC3 internally (which I think is not yet the case, right?)

In general it would obviously be best for as much of this Py8-specific logic as possible to live in the Pythia8 codebase as library functions, and hence will evolve and update "automatically" with Pythia features. We have a similar issue with the ATLAS Py8 interface. So I'll mention this also, next time we raise the idea of pushing most of the interfacing physics-logic into libPythia.

[Olivier Mattelaer (olivier-mattelaer)]

Hi Andy,

Yes indeed when that code was implemented we had to workaround an hepmc bug of the time.
And stephan was forcing to use a dedicated version of hepmc (that we likely still need/ship) in order for this to work.

Concerning the scomp/... flag, I have simply remove their inclusion within the hepmc file. Since I'm not aware of anyone using those information for computing merging scale uncertainty from post-processing the hepmc file. In any case for the people doing it (if any), this is as problematic as changing the name of those weights. So let's wait that someone complain here.

For hepmc3, if you want us to support it, the best is likely to do it within the cern school when we can sit together with a pythia8 author and see what needs to be done, in principle this is likely a pure change of configuration within pythia8 and, maybe, a bunch of check to adapt (the one checking that py8 produces the correct output file). But yes that file might be the complex part to handle...