STDHEP files larger than 4GB are corrupted

[Pavel Demin]

I've just tested several STDHEP files produced by MG+HERWIG6.

I've tested 50MB, 3.3GB and 4.0GB files.

The 50MB and 3.3GB files can be read without any problem.

The 4.0GB file can't be read.

In the 4.0GB file, already the 2nd block has a block ID number equal 2. Normally, it should be 4 (Event Header). The following blocks have completely wrong block ID numbers. So, I'd say that it's corrupted from the very beginning.

In some cases, adding the following compilation flag helps:

-D_FILE_OFFSET_BITS=64

Normally, the STDHEP library and maybe the PYTHIA and HERWIG interfaces should be compiled with this flag.

If you can tell me where to put this flag, I can redo the test.

[Paolo Torrielli (paolo-torrielli)]

Hi Pavel,
I think you should put that flag in MCatNLO-Utilities/StdHEP/mcfio/src/GNUMakefile,
in MCatNLO-Utilities/StdHEP/src/stdhep/GNUMakefile, recompile and copy the created
libraries libFmcfio.a and libstdhep.a into MCatNLO-Utilities/MCatNLO/lib.
Let me know if it works.
Cheers.
Paolo

[Pavel Demin (pavel-demin)]

Hi,

Why it's changed to invalid?

[Olivier Mattelaer (olivier-mattelaer)]

Looks like they were no activity on this anymore for a long time, so I close the bug.
Please re-open it if you want/need

[Paolo Torrielli (paolo-torrielli)]

Hi,
I don't know, I have the same question. Olivier could you explain?
Thanks.
Cheers.
Paolo

On Oct 14, 2013, at 6:04 PM, Pavel Demin <email address hidden> wrote:

> Hi,
>
> Why it's changed to invalid?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1218842
>
> Title:
> STDHEP files larger than 4GB are corrupted
>
> Status in The MadGraph Matrix Element Generator version 5:
> Invalid
>
> Bug description:
> I've just tested several STDHEP files produced by MG+HERWIG6.
>
> I've tested 50MB, 3.3GB and 4.0GB files.
>
> The 50MB and 3.3GB files can be read without any problem.
>
> The 4.0GB file can't be read.
>
> In the 4.0GB file, already the 2nd block has a block ID number equal
> 2. Normally, it should be 4 (Event Header). The following blocks have
> completely wrong block ID numbers. So, I'd say that it's corrupted
> from the very beginning.
>
> In some cases, adding the following compilation flag helps:
>
> -D_FILE_OFFSET_BITS=64
>
> Normally, the STDHEP library and maybe the PYTHIA and HERWIG
> interfaces should be compiled with this flag.
>
> If you can tell me where to put this flag, I can redo the test.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/madgraph5/+bug/1218842/+subscriptions

[Pavel Demin (pavel-demin)]

I had a busy month in September and could not work on this issue. I'll try to redo my tests in the coming days.

[Olivier Mattelaer (olivier-mattelaer)]

Ok I re-open the bug then.

Thanks,

Olivier

[Pavel Demin (pavel-demin)]

I've just looked at the mcfio/stdhep code includedd with the pythia-pgs package in MG5.

The code in pythia-pgs/src/pythia.f calls the following chain of functions to write an event:

 1. stdxwrt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwrt.F
 2. stdxwevt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwevt.F
 3. mcfio_block defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src/mcfio_Block.c

Then he mcfio_block function uses the xdr_getpos and xdr_setpos functions from the XDR library. These functions are used to get and set postion in the output file.

The problem is that these functions operate on 32-bit values and return 32-bit values. When the position in file reaches the 4GB limit, the 32-bit position overflows and the mcfio/sdthep librarry starts to write events in the very beginning of the file and thus corrupts the file header.

So, my conclusion is that the mcfio/stdhep library used by the pythia-pgs/src/pythia.f program can't work with files larger than 4GB.

[Olivier Mattelaer (olivier-mattelaer)]

Thanks a lot for the investigation.

Cheers,

Olivier
On Nov 27, 2013, at 10:29 PM, Pavel Demin <email address hidden> wrote:

> I've just looked at the mcfio/stdhep code includedd with the pythia-pgs
> package in MG5.
>
> The code in pythia-pgs/src/pythia.f calls the following chain of
> functions to write an event:
>
> 1. stdxwrt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwrt.F
> 2. stdxwevt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwevt.F
> 3. mcfio_block defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src/mcfio_Block.c
>
> Then he mcfio_block function uses the xdr_getpos and xdr_setpos
> functions from the XDR library. These functions are used to get and set
> postion in the output file.
>
> The problem is that these functions operate on 32-bit values and return
> 32-bit values. When the position in file reaches the 4GB limit, the
> 32-bit position overflows and the mcfio/sdthep librarry starts to write
> events in the very beginning of the file and thus corrupts the file
> header.
>
> So, my conclusion is that the mcfio/stdhep library used by the pythia-
> pgs/src/pythia.f program can't work with files larger than 4GB.
>
> --
> You received this bug notification because you are a member of MadTeam,
> which is subscribed to MadGraph5.
> https://bugs.launchpad.net/bugs/1218842
>
> Title:
> STDHEP files larger than 4GB are corrupted
>
> Status in The MadGraph Matrix Element Generator version 5:
> Confirmed
>
> Bug description:
> I've just tested several STDHEP files produced by MG+HERWIG6.
>
> I've tested 50MB, 3.3GB and 4.0GB files.
>
> The 50MB and 3.3GB files can be read without any problem.
>
> The 4.0GB file can't be read.
>
> In the 4.0GB file, already the 2nd block has a block ID number equal
> 2. Normally, it should be 4 (Event Header). The following blocks have
> completely wrong block ID numbers. So, I'd say that it's corrupted
> from the very beginning.
>
> In some cases, adding the following compilation flag helps:
>
> -D_FILE_OFFSET_BITS=64
>
> Normally, the STDHEP library and maybe the PYTHIA and HERWIG
> interfaces should be compiled with this flag.
>
> If you can tell me where to put this flag, I can redo the test.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/madgraph5/+bug/1218842/+subscriptions

[Paolo Torrielli (paolo-torrielli)]

Yes, thanks very much.
Cheers.
Paolo

On Nov 28, 2013, at 12:20 AM, Olivier Mattelaer <email address hidden> wrote:

> Thanks a lot for the investigation.
>
> Cheers,
>
> Olivier
> On Nov 27, 2013, at 10:29 PM, Pavel Demin <email address hidden> wrote:
>
>> I've just looked at the mcfio/stdhep code includedd with the pythia-pgs
>> package in MG5.
>>
>> The code in pythia-pgs/src/pythia.f calls the following chain of
>> functions to write an event:
>>
>> 1. stdxwrt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwrt.F
>> 2. stdxwevt defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/src/stdhep/stdxwevt.F
>> 3. mcfio_block defined in pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src/mcfio_Block.c
>>
>> Then he mcfio_block function uses the xdr_getpos and xdr_setpos
>> functions from the XDR library. These functions are used to get and set
>> postion in the output file.
>>
>> The problem is that these functions operate on 32-bit values and return
>> 32-bit values. When the position in file reaches the 4GB limit, the
>> 32-bit position overflows and the mcfio/sdthep librarry starts to write
>> events in the very beginning of the file and thus corrupts the file
>> header.
>>
>> So, my conclusion is that the mcfio/stdhep library used by the pythia-
>> pgs/src/pythia.f program can't work with files larger than 4GB.
>>
>> --
>> You received this bug notification because you are a member of MadTeam,
>> which is subscribed to MadGraph5.
>> https://bugs.launchpad.net/bugs/1218842
>>
>> Title:
>> STDHEP files larger than 4GB are corrupted
>>
>> Status in The MadGraph Matrix Element Generator version 5:
>> Confirmed
>>
>> Bug description:
>> I've just tested several STDHEP files produced by MG+HERWIG6.
>>
>> I've tested 50MB, 3.3GB and 4.0GB files.
>>
>> The 50MB and 3.3GB files can be read without any problem.
>>
>> The 4.0GB file can't be read.
>>
>> In the 4.0GB file, already the 2nd block has a block ID number equal
>> 2. Normally, it should be 4 (Event Header). The following blocks have
>> completely wrong block ID numbers. So, I'd say that it's corrupted
>> from the very beginning.
>>
>> In some cases, adding the following compilation flag helps:
>>
>> -D_FILE_OFFSET_BITS=64
>>
>> Normally, the STDHEP library and maybe the PYTHIA and HERWIG
>> interfaces should be compiled with this flag.
>>
>> If you can tell me where to put this flag, I can redo the test.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/madgraph5/+bug/1218842/+subscriptions
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1218842
>
> Title:
> STDHEP files larger than 4GB are corrupted
>
> Status in The MadGraph Matrix Element Generator version 5:
> Confirmed
>
> Bug description:
> I've just tested several STDHEP files produced by MG+HERWIG6.
>
> I've tested 50MB, 3.3GB and 4.0GB files.
>
> The 50MB and 3.3GB files can be read without any problem.
>
> The 4.0GB file can't be read.
>
> In the 4.0GB file, already the 2nd block has a block ID number equal
> 2. Normally, it should be 4 (Event Header). The following block...

[Pavel Demin (pavel-demin)]

As a fast fix, I've replaced xdr_getpos/xdr_setpos with ftello/fseeko in some of the mcfio files and set the type of the position variables to off_t.

Now I can generate a 5GB STDHEP file and it does not get corrupted.

Here is a list of modified files in pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src:

mcfio_Block.c
mcfio_Direct.c
mcfio_Util1.c
mcf_NTuIOFiles.c
mcf_xdr.h

You can install the modified files with the following commands:

cd pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files.tgz
tar -vzxf pythia-pgs-mcfio-files.tgz

Could you, please commit these modifications to the pythia-pgs repository?

[Olivier Mattelaer (olivier-mattelaer)]

Excellent :-)

I will push this fix thanks so so much.

Olivier
On Nov 28, 2013, at 1:06 PM, Pavel Demin <email address hidden> wrote:

> As a fast fix, I've replaced xdr_getpos/xdr_setpos with ftello/fseeko in
> some of the mcfio files and set the type of the position variables to
> off_t.
>
> Now I can generate a 5GB STDHEP file and it does not get corrupted.
>
> Here is a list of modified files in pythia-pgs/libraries/PGS4/src
> /stdhep-dir/mcfio/src:
>
> mcfio_Block.c
> mcfio_Direct.c
> mcfio_Util1.c
> mcf_NTuIOFiles.c
> mcf_xdr.h
>
> You can install the modified files with the following commands:
>
> cd pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src
> wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files.tgz
> tar -vzxf pythia-pgs-mcfio-files.tgz
>
> Could you, please commit these modifications to the pythia-pgs
> repository?
>
> --
> You received this bug notification because you are a member of MadTeam,
> which is subscribed to MadGraph5.
> https://bugs.launchpad.net/bugs/1218842
>
> Title:
> STDHEP files larger than 4GB are corrupted
>
> Status in The MadGraph Matrix Element Generator version 5:
> Confirmed
>
> Bug description:
> I've just tested several STDHEP files produced by MG+HERWIG6.
>
> I've tested 50MB, 3.3GB and 4.0GB files.
>
> The 50MB and 3.3GB files can be read without any problem.
>
> The 4.0GB file can't be read.
>
> In the 4.0GB file, already the 2nd block has a block ID number equal
> 2. Normally, it should be 4 (Event Header). The following blocks have
> completely wrong block ID numbers. So, I'd say that it's corrupted
> from the very beginning.
>
> In some cases, adding the following compilation flag helps:
>
> -D_FILE_OFFSET_BITS=64
>
> Normally, the STDHEP library and maybe the PYTHIA and HERWIG
> interfaces should be compiled with this flag.
>
> If you can tell me where to put this flag, I can redo the test.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/madgraph5/+bug/1218842/+subscriptions

[Olivier Mattelaer (olivier-mattelaer)]

Ok the patch is pushed.
and should be distributed with any new attempt to install the code.
(But for the short period between the time where the UCL server will be up again, and the update of that cluster)

Cheers and thanks again,

Olivier

On Nov 28, 2013, at 11:50 PM, Olivier Mattelaer <email address hidden> wrote:

> Excellent :-)
>
> I will push this fix thanks so so much.
>
> Olivier
> On Nov 28, 2013, at 1:06 PM, Pavel Demin <email address hidden> wrote:
>
>> As a fast fix, I've replaced xdr_getpos/xdr_setpos with ftello/fseeko in
>> some of the mcfio files and set the type of the position variables to
>> off_t.
>>
>> Now I can generate a 5GB STDHEP file and it does not get corrupted.
>>
>> Here is a list of modified files in pythia-pgs/libraries/PGS4/src
>> /stdhep-dir/mcfio/src:
>>
>> mcfio_Block.c
>> mcfio_Direct.c
>> mcfio_Util1.c
>> mcf_NTuIOFiles.c
>> mcf_xdr.h
>>
>> You can install the modified files with the following commands:
>>
>> cd pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src
>> wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files.tgz
>> tar -vzxf pythia-pgs-mcfio-files.tgz
>>
>> Could you, please commit these modifications to the pythia-pgs
>> repository?
>>
>> --
>> You received this bug notification because you are a member of MadTeam,
>> which is subscribed to MadGraph5.
>> https://bugs.launchpad.net/bugs/1218842
>>
>> Title:
>> STDHEP files larger than 4GB are corrupted
>>
>> Status in The MadGraph Matrix Element Generator version 5:
>> Confirmed
>>
>> Bug description:
>> I've just tested several STDHEP files produced by MG+HERWIG6.
>>
>> I've tested 50MB, 3.3GB and 4.0GB files.
>>
>> The 50MB and 3.3GB files can be read without any problem.
>>
>> The 4.0GB file can't be read.
>>
>> In the 4.0GB file, already the 2nd block has a block ID number equal
>> 2. Normally, it should be 4 (Event Header). The following blocks have
>> completely wrong block ID numbers. So, I'd say that it's corrupted
>> from the very beginning.
>>
>> In some cases, adding the following compilation flag helps:
>>
>> -D_FILE_OFFSET_BITS=64
>>
>> Normally, the STDHEP library and maybe the PYTHIA and HERWIG
>> interfaces should be compiled with this flag.
>>
>> If you can tell me where to put this flag, I can redo the test.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/madgraph5/+bug/1218842/+subscriptions
>
> --
> You received this bug notification because you are a member of MadTeam,
> which is subscribed to MadGraph5.
> https://bugs.launchpad.net/bugs/1218842
>
> Title:
> STDHEP files larger than 4GB are corrupted
>
> Status in The MadGraph Matrix Element Generator version 5:
> Confirmed
>
> Bug description:
> I've just tested several STDHEP files produced by MG+HERWIG6.
>
> I've tested 50MB, 3.3GB and 4.0GB files.
>
> The 50MB and 3.3GB files can be read without any problem.
>
> The 4.0GB file can't be read.
>
> In the 4.0GB file, already the 2nd block has a block ID number equal
> 2. Normally, it should be 4 (Event Header). The following blocks have
> completely wrong block ID numbers. So, I'd say that it's corrupted
> from the very be...

[Pavel Demin (pavel-demin)]

Hi,

Could you, please change the status of this bug to something like "Unsolved"?

Several MadGraph users noticed that large files written by the patched STDHEP library can't be read back by this same library:

https://answers.launchpad.net/mg5amcnlo/+question/244320

The programs that use the STDHEP library for reading the STDHEP files still can't read files larger than 4GB. These programs include PGS and hep2lhe.

While trying to find the source of this problem and looking more carefully at the STDHEP format, I've realized that the problem is with the format and not just the library.

This format has structures like "Event Header" and "Event Table". Event Header contains a list of positions of blocks and Event Table contains a list of positions of events. Both positions are encoded as 32-bit unsigned integers and are written to the file.

I'll try to change the 32-bit blocks positions and events positions to 64-bit values and create new version numbers for the "Event Header" and "Event Table" structures.

Cheers,

Pavel

[Pavel Demin (pavel-demin)]

Hi,

Here is a new patch.

I've added version 2.00 for Event Header and version 3.00 for Event Table.

These new versions store blocks and events positions as 64-bit values.

Improvements:
 - Pythia produces STDHEP files larger than 4GB
 - PGS and hep2lhe read STDHEP files larger than 4GB
 - PGS and hep2lhe read both new and previous format versions

Disadvantages:
 - ExRootAnalysis, Delphes3 and MadGraph5 should be adapted for this new format version

Actually, I have already a new ExRootAnalysis version that can read both new and previous format versions.

Please, let me know if this patch seems OK for you.

Here are the commands to apply this patch for MG5:

cd pythia-pgs/libraries/PGS4/src/stdhep-dir/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files-v2.tgz
tar -zxf pythia-pgs-mcfio-files-v2.tgz

and here are the commands to apply this patch for MG5_aMC:

cd vendor/StdHEP/mcfio/src
wget http://cp3.irmp.ucl.ac.be/downloads/pythia-pgs-mcfio-files-v2.tgz
tar -zxf pythia-pgs-mcfio-files-v2.tgz

Cheers,

Pavel

[Olivier Mattelaer (olivier-mattelaer)]

Thanks a lot for this.

- The fix is pushed/published for pythia-pgs. (a new install pythia-pgs should work).
- Your fix for ExRootAnalysis should be published as well.
- The fix inside MG5_aMC is pushed in 2.1.2, and will be part of the next release.

Thanks so much,

Olivier

[Rakhi Mahbubani (r-mahbubani)]

Hi Pavel and Olivier,

After installing this patch, I've been having issues with pythia consistently crashing approximately half-way through showering my events with the error message "mcfio_Block: unable to encode or decode block I.D.101". I don't think I was having this issue before the patch. My resultant hep file has a size of 2.0 GB. Please can you help?

Cheers,

Rakhi

[Pavel Demin (pavel-demin)]

Hi Rakhi,

Please provide a detailed list of commands (starting from downloading MadGraph and patching) that would allow me to reproduce your problem.

cheers,

Pavel

[Rakhi Mahbubani (r-mahbubani)]

Dear Pavel,

Sorry, I totally missed this: for some reason all launchpad's emails are landing in my spam. Would my process banner suffice, or are you looking for additional information?

Cheers,

Rakhi