StdHep of pythia-pgs 2.2.1 or higher cannot be converted to HepMC

[Matthias Schlaffer]

The *.hep file produced by pythia-pgs v2.2.1 or higher cannot be converted to HepMC format by convertStdHep from http://cepa.fnal.gov/psm/stdhep/

When running the conversion, only an empty HepMC file is created and the output on the screen is:

 input file: events.hep
 output file: events.hepmc
 StdHepXdrReadOpen: successfully opened input stream 1
          title: PYTHIA file
          date: Mon Jun 16 16:13:34 2014

                    351 events
                    8 blocks per event
 mcfio_NextEvent: XDR Error decoding the EventTable
     StdHepXdrRead: unrecognized status - stop
 end of file after 0 events
 end of file after 0 events
0 events read
          StdHepXdrEnd: 149 words i/o with 14093 efficiency

[Olivier Mattelaer (olivier-mattelaer)]

We had to change the stdhep format in order to allow file larger than 4G. Since this was a limitation in a number of application.
I guess that the converter that you point need to be update to support the new format.

Pavel can give more information on that subject.

Cheers,

Olivier

[Pavel Demin (pavel-demin)]

Please, provide a detailed list of commands needed to download and build the convertStdHep program.

[Matthias Schlaffer (matthias-schlaffer)]

Hi Olivier,

I saw this comment in the changelog, however according to the update notes this change comes only with version 2.3.0 and not with version 2.2.1. So I expected version 2.2.1 to work and was a bit confused.

Best,
Matthias

[Matthias Schlaffer (matthias-schlaffer)]

Hi Pavel,

downloading and building goes as follows:

wget http://cepa.fnal.gov/psm/stdhep/dist/convertStdHep-0.02.00.tar.gz
tar -zxf convertStdHep-0.02.00.tar.gz

# in convertStdHep/src/convertStdHep.cc replace IO_Ascii by IO_AsciiParticles (two occurrences)
# and add '#include <cstring>'

cd convertStdHep
mkdir install build
cd build
../configure --prefix=/path/to/convertStdHep/install --with-HepMC=/path/to/hepMC/install #here this is HepMC v2.06.09
make
make install

then you can run it with ./convertStdHep/install/bin/convertStdHep -i inputfile.hep -o outputfile.hepmc

I did not check yet if the changes in convertStdHep.cc affect the converted file (which still looks reasonable), but they were necessary to build the program.

Best,
Matthias

[Emmanuel Stamou (miscman)]

Hi Matthias,

thanks for posting the bug and the way to compile convertStdHep. Very helpful. I'm having the same issue with the Herwig hep file.

One question, what do you mean above with:

"the converted file (which still looks reasonable)" ?

Isn't also your problem that the converted file is empty? So what looks reasonable?

Best,
emmanuel

[Matthias Schlaffer (matthias-schlaffer)]

Hi Emmanuel,

when I used an earlier version of pythia-pgs and compiled convertStdHep according to the steps mentioned above I did not get an empty file but one which contains events. This is what I meant by "looks reasonable". I hope this helps.

Best,
Matthias

[Emmanuel Stamou (miscman)]

Hi Pavel, hi Olivier,

this is a very related question/observation so I did not want to start a new thread.

I have the feeling that something went wrong when you had to change the StdHep format. Under vendor/StdHEP/ in the Madgraph installation I see that even though version v5_06_01 is quoted, the StdHep code is modified. This is fine, but in my case it seems that the standard commands provided by StdHep to read events from the hep file lead to errors, i.e. one cannot read the event files anymore.

For instance the program vendor/StdHEP/example/testCInOut.c cannot read the HERWIG hep file produced by the current version of MG5 - StdHepXdrRead() does not read the event. The same thing worked in prior versions of MG5, I explicitly checked.

This is also why the conversion program no longer works. Is this behaviour intended or possible to evade?

Best
emmanuel

[Pavel Demin (pavel-demin)]

Hi Matthias,

Sorry for answering that late.

I've just had a look at the convertStdHep-0.02.00.tar.gz file and it looks like the same patch that we applied to the mcfio library that comes with MadGraph should work for convertStdHep.

Following you instructions, I've downloaded convertStdHep-0.02.00.tar.gz, modified convertStdHep/src/convertStdHep.cc and applied the patch that I posted in

https://bugs.launchpad.net/mg5amcnlo/+bug/1218842/comments/14

The resulting convertStdHep program works with STDHEP files generated by the latest version of MadGraph5/pythia-pgs.

Here are the commands to apply this patch:

cd convertStdHep/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files-v2.tgz
tar -zxf pythia-pgs-mcfio-files-v2.tgz

Regards,

Pavel

[Matthias Schlaffer (matthias-schlaffer)]

Hi Pavel,

Thank you very much for the fix. That solved it!

Best,
Matthias

[Reggie Bain (rab59)]

Hi Pavel/Matthias,

I am trying to convert .hepmc on .hep files created by pythia-pgs 2.4.0 installed for Madgraph 4.5.2. I having followed the steps above (except those dealing with MG5 specifically at the end) and am having the same problem as described above when using convertStdHep. I am given the error:

 input file: myrun_pythia_events.hep
 output file: myrun_pythia_events.hepmc
 StdHepXdrReadOpen: successfully opened input stream 1
          title: PYTHIA file
          date: Tue Sep 23 14:14:25 2014

                    1002 events
                    8 blocks per event
 mcfio_NextEvent: XDR Error decoding the EventTable
     StdHepXdrRead: unrecognized status - stop
 end of file after 0 events
0 events read

Do these patches not also work for Madgraph 4 pythia-pgs? Do I need to use an older version of pythia-pgs? If so where can I download older versions?

Thank you,

Reggie

[Pavel Demin (pavel-demin)]

Hi Reggie,

Could you post a list of commands that you are running? It'll allow me to reproduce your problem.

Normally, the patch that I posted in

https://bugs.launchpad.net/mg5amcnlo/+bug/1218842/comments/14

should work for MG4 as well.

Have you tried the following commands?

cd pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files-v2.tgz
tar -zxf pythia-pgs-mcfio-files-v2.tgz

and

cd convertStdHep/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files-v2.tgz
tar -zxf pythia-pgs-mcfio-files-v2.tgz

After applying these patches you need to rebuild pythia-pgs and convertStdHep.

Regards,

Pavel

[Reggie Bain (rab59)]

Hi Pavel,

Yes, I tried those two but after going back I may have another issue as well. Before I was using pythia-pgs with MadOnia but I need to migrate to using full Madgraph 4.5.2. I am getting some errors when building pythia-pgs. Here are the commands I followed according to https://bugs.launchpad.net/pythia-pgs-for-mg/+bug/1202197.

After commenting out lines with: -fno-second-underscore

cd /pathToMG/MG_ME_V4.5.2/pythia-pgs/src/
make libpythia
make

gfortran -O -fno-automatic -I../libraries/PGS4/src -o pythia pythia.o ME2pythia.o getjet.o ktclusdble.o pgs_ranmar.o pydata.o -L../libraries/PGS4/lib -L../libraries/pylib/lib -L../libraries/lhapdf/lib -ltauola -lpythiaext -lstdhep -lFmcfio -lLHAPDFlocal
gfortran -O -fno-automatic -I../libraries/PGS4/src -o hep2lhe hep2lhe.o ME2pythia.o getjet.o ktclusdble.o pgs_ranmar.o pydata.o -L../libraries/PGS4/lib -L../libraries/pylib/lib -L../libraries/lhapdf/lib -ltauola -lpythiaext -lstdhep -lFmcfio -lLHAPDFlocal
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
make: *** [hep2lhe] Error 1

And of course when just trying 'make' in /pythia-pgs/ I get:

gfortran -O -fno-automatic -I../libraries/PGS4/src -o pythia pythia.o ME2pythia.o getjet.o ktclusdble.o pgs_ranmar.o pydata.o -L../libraries/PGS4/lib -L../libraries/pylib/lib -L../libraries/lhapdf/lib -ltauola -lpythiaext -lstdhep -lFmcfio -lLHAPDFlocal
gfortran -O -fno-automatic -I../libraries/PGS4/src -o hep2lhe hep2lhe.o ME2pythia.o getjet.o ktclusdble.o pgs_ranmar.o pydata.o -L../libraries/PGS4/lib -L../libraries/pylib/lib -L../libraries/lhapdf/lib -ltauola -lpythiaext -lstdhep -lFmcfio -lLHAPDFlocal
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
make[1]: *** [hep2lhe] Error 1
make[1]: Leaving directory `/home/rab59/rivet/local/MG_ME_V4.5.2/pythia-pgs/src'
make: *** [all] Error 2

Note that I'm using pythia-pgs 2.4.0 (that's the only one I could find to download) and MG 4.5.2. My system is:

Linux version 3.14.17-100.fc19.x86_64 (<email address hidden>) (gcc version 4.8.3 20140624 (Red Hat 4.8.3-1) (GCC) )

I followed your directions above and installed the patch for convertStdHep and that seems to build nicely.

Thank you again,

Reggie

[Reggie Bain (rab59)]

Hi Pavel,

It seems that even with this build error I mentioned above, MG4 can still make a .hep file (but of course it can't convert it to .lhe). When applying the patches outlined above and in the threads you suggested...convertStdHep does run...however I was under the impression that it would convert the .hep into a traditional .hepmc format that looks like the following:

HepMC::Version 2.06.06
HepMC::IO_GenEvent-START_EVENT_LISTING
E 2 -1 -1.0000000000e+00 -1.0000000000e+00 -1.0000000000e+00 0 0 347 1 2 0 1 1.0000000000e+00
N 1 "0"
U GEV MM
C 7.0834789202e+02 7.0834789202e+02
F 21 21 1.2748621498e-02 1.8666249359e-01 4.5383731768e+01 6.5904825914e+00 2.9607314395e-01 0 0
V -1 0 0 0 0 0 1 2 0
P 1 2212 0 0 3.9999998900e+03 4.0000000000e+03 9.3827000000e-01 4 0 0 -1 0
P 3 1 -2.7727745346e-01 1.1927342005e-01 7.8824800075e+02 7.8824805854e+02 0 3 0 0 -3 0
P 188 2203 2.7727745346e-01 -1.1927342005e-01 1.7901503639e+03 1.7901505555e+03 7.7133000000e-01 2 0 0 -90 0
V -2 0 0 0 0 0 1 3 0

My goal is to use this format to input to Rivet for analysis and plotting. The format I get out of convertStdHep looks as follows:

0 Run HepMC::IO_AsciiParticles eye-readable events output
# HepMC::Version 2.06.06
  # stat pdg moth1 px py pz energy mass eta
0 Event
0 particles
1 Event
905 particles
   1 3 2212 0 0.00e+00 0.00e+00 4.00e+03 4.00e+03 0.938 999
   2 3 2212 0 0.00e+00 0.00e+00 -4.00e+03 4.00e+03 0.938 -999
   3 3 21 1 1.80e+00 -2.85e+00 6.42e+01 6.43e+01 0 3.64
   4 3 2 2 3.41e-01 -2.80e-02 -2.30e+03 2.30e+03 0 -9.51
   5 3 21 3 2.01e+00 -1.88e+00 5.03e+01 5.03e+01 0 3.6
   6 3 21 4 -4.92e+00 1.32e+01 -5.32e+01 5.51e+01 0 -2.04
   7 3 21 5 -4.17e+01 2.59e+01 2.53e+01 5.52e+01 0 0.494
   8 3 443 5 3.88e+01 -1.46e+01 -2.82e+01 5.02e+01 3 -0.637
   9 0 0 0 0.00e+00 0.00e+00 0.00e+00 0.00e+00 0 0

When run with rivet...this format yields:

Failed to initialise using event file '/home/rab59/rivet/convertStdHep/testp6.hepmc'... exiting

How can I get convertStdHep to output the (what I thought to be) more traditional hepmc style?

Thank you again,

Reggie

[Pavel Demin (pavel-demin)]

Hi Reggie,

Looking at one of the Pythia 8 examples (examples/main41.cc), I'd suggest to replace

IO_AsciiParticles

with

IO_GenEvent

in convertStdHep/src/convertStdHep.cc.

Here are the commands that I used in a directory where I previously built convertStdHep following the instructions from https://bugs.launchpad.net/pythia-pgs-for-mg/+bug/1334366/comments/4

cd convertStdHep
sed -i 's/IO_AsciiParticles/IO_GenEvent/' src/convertStdHep.cc
cd build
make
make install

With this modification, convertStdHep outputs HepMC in the format that you described in your last comment.

Regards,

Pavel

[Reggie Bain (rab59)]

Again, thank you so much for the prompt feedback! When I make that replacement it seems that when I make this adjustment I get the following error:

configure: error: source directory already configured; run "make distclean" there first
make: *** [config.status] Error 1

When I ran 'make distclean' in convertStdHep/src I got the following output:

[rab59@grads-c3 src]$ make distclean
cd .. && make am--refresh
make[1]: Entering directory `/home/rab59/rivet/convertStdHep'
/bin/sh ./config.status --recheck
running /bin/sh ./configure --prefix=/home/rab59/rivet/convertStdHep/install --with-HepMC=/home/rab59/rivet/local --no-create --no-recursion
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether ln -s works... yes
checking for ranlib... ranlib
checking for cl... no
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking for cl... no
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for g77... g77
checking whether we are using the GNU Fortran 77 compiler... yes
checking whether g77 accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
configure: creating ./config.status
 /bin/sh ./config.status
config.status: creating Makefile
config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting
config.status: creating convertStdHep/Makefile
config.status: WARNING: 'convertStdHep/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating src/Makefile
config.status: WARNING: 'src/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating test/Makefile
config.status: WARNING: 'test/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating test/testConvertStdHep.sh
config.status: creating convertStdHep/defs.h
config.status: executing depfiles commands
make[1]: Leaving directory `/home/rab59/rivet/convertStdHep'
make[1]: Entering directory `/home/rab59/rivet/convertStdHep'
make[1]: Leaving directory `/home/rab59/rivet/convertStdHep'
make[1]: Entering directory `/home/rab59/rivet/convertStdHep'
make[1]: Leaving directory `/home/rab59/rivet/convertStdHep'
test -z "convertStdHep" || rm -f convertStdHep
rm -f *.o
rm -f *.tab.c
test -z "" || rm -f
rm -f TAGS ID GTAGS GRTAGS GSYMS GPATH tags
rm -rf ./.deps
rm -f Makefile

However, when going back to convertStdHep/build and running 'make' I still go...

[Pavel Demin (pavel-demin)]

Hi Reggie,

What command outputs the error message? Is it 'configure'?

Normally, you don't need to rerun 'configure'.

To rebuild convertStdHep after modifying convertStdHep.cc you can run 'make'.

I've posted the exact commands that I used to rebuild convertStdHep in

https://bugs.launchpad.net/pythia-pgs-for-mg/+bug/1334366/comments/14

Alternatively, you can try to remove and recreate your build directory and the rerun 'configure', etc.

Regards,

Pavel