Configuration writing to a read-only file system in 2.5.1

[Zachary Marshall]

Hello,

In 2.5.1, we can run:

$MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC

generate p p > t t~ [QCD]
output

where the installation directory is on a read-only file system, but `pwd` is read/write (e.g. a scratch area or tmp directory). This now asks about which programs to install, and then attempts two write a configuration file back to the read-only area:

set golem ''
save options --auto
save configuration file to
/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
Command "output tt" interrupted with error:
IOError : [Errno 30] Read-only file system:
'/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt'

I believe this is because of a change in

madgraph/interface/extended_cmd.py

which now tries to save a configuration file via the write_configuration function when you try to do almost anything. I think, since we have the dependencies set to save locally, the correct behavior would be to make a locally directory and write there (or simply not update the config card at all).

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

The problem is that at the first time the code is run at NLO (more exactly the first time a loop is requested) we check the status of the ninja package.
If that option is on default and not installed, then we ask a question on which set of program need to be installed to perform the reduction of the loop.

One potential solution would be to not ask the question at all if you do not have writeacces.
Can you test that? The patch is written at the end of the message.

A work around is to ensure that all the reduction tools that you do not want are set to None in the
$MADGRAPH_INSTALLATION_DIR/input/mg5_configuration.txt

Cheers,

Olivier

PS: I would strongly suggest to not use 2.5.1 since 2.5.2 contains a lot of important (but not critical) bug fixing. One of those is that some log file are HUGE in 2.5.1

=== modified file 'madgraph/interface/loop_interface.py'
--- madgraph/interface/loop_interface.py 2016-09-01 10:26:58 +0000
+++ madgraph/interface/loop_interface.py 2017-01-19 08:29:15 +0000
@@ -23,7 +23,7 @@
 import re

 import madgraph
-from madgraph import MG4DIR, MG5DIR, MadGraph5Error
+from madgraph import MG4DIR, MG5DIR, MadGraph5Error, ReadWrite
 import madgraph.interface.madgraph_interface as mg_interface
 import madgraph.interface.extended_cmd as cmd
 import madgraph.interface.launch_ext_program as launch_ext
@@ -500,6 +500,8 @@
         # Check if first time:
         if (opt['ninja'] is None) or (os.path.isfile(pjoin(MG5DIR, opt['ninja'],'libninja.a'))):
             return
+ if not ReadWrite:
+ return

         logger.info("First output using loop matrix-elements has been detected. Now asking for loop reduction:", '$MG:color:BLACK')
         to_install = self.ask('install', '0', ask_class=AskLoopInstaller, timeout=300,

> On Jan 19, 2017, at 02:26, Zachary Marshall <email address hidden> wrote:
>
> Public bug reported:
>
> Hello,
>
> In 2.5.1, we can run:
>
> $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC
>
> generate p p > t t~ [QCD]
> output
>
> where the installation directory is on a read-only file system, but
> `pwd` is read/write (e.g. a scratch area or tmp directory). This now
> asks about which programs to install, and then attempts two write a
> configuration file back to the read-only area:
>
> set golem ''
> save options --auto
> save configuration file to
> /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
> Command "output tt" interrupted with error:
> IOError : [Errno 30] Read-only file system:
> '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt'
>
> I believe this is because of a change in
>
> madgraph/interface/extended_cmd.py
>
> which now tries to save a configuration file via the write_configuration
> function when you try to do almost anything. I think, since we have the
> dependencies set to save locally, the correct behavior would be to make
> a locally directory and write there (or simply not update the config
> ...

[Zachary Marshall (zach-marshall)]

Thanks Olivier, that does indeed solve the first problem. It seems as though the system continues to try to install the external dependencies in the MadGraph area that should be read-only, though -- should these not count as dependencies to be installed externally?

Thanks,
Zach

[Zachary Marshall (zach-marshall)]

Hi Olivier (and Valentin),

Sorry, we just hit this again with an LO setup that wasn't trying to install any dependencies:

Traceback (most recent call last):
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/bin/mg5_aMC", line 141, in <module>
    cmd_line = interface.MasterCmd(mgme_dir = options.mgme_dir)
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/madgraph/interface/master_interface.py", line 588, in __init__
    self.cmd.__init__(self, *args, **opt)
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/madgraph/interface/madgraph_interface.py", line 2872, in __init__
    shutil.copy(make_opts_source, make_opts)
  File "/afs/cern.ch/sw/lcg/external/Python/2.7.3/x86_64-slc6-gcc47-opt/lib/python2.7/shutil.py", line 117, in copy
    copyfile(src, dst)
  File "/afs/cern.ch/sw/lcg/external/Python/2.7.3/x86_64-slc6-gcc47-opt/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/Template/LO/Source/make_opts'

The problem is worse in MadGraph 2.5.2, where literally launching the executable on lxplus causes this crash. You can try yourself at CERN if you would like:

/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/bin/mg5_aMC

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

This file needs to be created at the first time the code is executed.
If you run the code once when you install the code, this should solve the problem.
If you do not want to run, you can just copy the file
Template/LO/Source/.make_opts to Template/LO/Source/make_opts
Cheers,

Olivier

[Zachary Marshall (zach-marshall)]

Hi Olivier,

After those patches, we still have an issue here in MG5_aMC 2.5.2. The output is:

set golem ''
save options --auto
save configuration file to /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas.1/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt

and seems to come from L579 of madgraph/interface/loop_interface.py:

                self.exec_cmd('save options %s' % key)

Presumably this should also be protected against a read-only file system?

Our latest installation is here:
/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas.1/x86_64-slc6-gcc47-opt/

and you can try running yourself to see the same on lxplus, if you would like.

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Can you attach the debug file? Not sure to understand what you are doing to have such set command activated.

For the lxplus access, I do not have access on those machine right now (but I guess I can ask next week to have such access if needed —I’m not at cern for the moment--)

Cheers,

Olivier

> On 16 Feb 2017, at 19:06, Zachary Marshall <email address hidden> wrote:
>
> Hi Olivier,
>
> After those patches, we still have an issue here in MG5_aMC 2.5.2. The
> output is:
>
> set golem ''
> save options --auto
> save configuration file to /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas.1/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
>
> and seems to come from L579 of madgraph/interface/loop_interface.py:
>
> self.exec_cmd('save options %s' % key)
>
> Presumably this should also be protected against a read-only file
> system?
>
> Our latest installation is here:
> /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas.1/x86_64-slc6-gcc47-opt/
>
> and you can try running yourself to see the same on lxplus, if you would
> like.
>
> Thanks,
> Zach
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1657615
>
> Title:
> Configuration writing to a read-only file system in 2.5.1
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Hello,
>
> In 2.5.1, we can run:
>
> $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC
>
> generate p p > t t~ [QCD]
> output
>
> where the installation directory is on a read-only file system, but
> `pwd` is read/write (e.g. a scratch area or tmp directory). This now
> asks about which programs to install, and then attempts two write a
> configuration file back to the read-only area:
>
> set golem ''
> save options --auto
> save configuration file to
> /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
> Command "output tt" interrupted with error:
> IOError : [Errno 30] Read-only file system:
> '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt'
>
> I believe this is because of a change in
>
> madgraph/interface/extended_cmd.py
>
> which now tries to save a configuration file via the
> write_configuration function when you try to do almost anything. I
> think, since we have the dependencies set to save locally, the correct
> behavior would be to make a locally directory and write there (or
> simply not update the config card at all).
>
> Thanks,
> Zach
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mg5amcnlo/+bug/1657615/+subscriptions

[Zachary Marshall (zach-marshall)]

Sure, it's attached

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

So the problem is that the path where you indicate that the ninja library is available does not exists.
In that case the code ask if you want to install it or not (and ask if you want to install other library like collier/golem/…)

The first question is do you want to include such library?
But if you do not want to, then you can set in the configuration file the path to ninja to empty (or None)
And this will solve the problem.

In general, I would recommend to run one LO and one NLO process when you install your process such that you are ensure
That all optional dependencies/side effects are correctly handle by the code itself.

Cheers,

Olivier

> On 17 Feb 2017, at 00:56, Zachary Marshall <email address hidden> wrote:
>
> Sure, it's attached
>
> ** Attachment added: "MG5_debug"
> https://bugs.launchpad.net/mg5amcnlo/+bug/1657615/+attachment/4820500/+files/MG5_debug
>
> --
> You received this bug notification because you are subscribed to
> MadGraph5_aMC@NLO.
> https://bugs.launchpad.net/bugs/1657615
>
> Title:
> Configuration writing to a read-only file system in 2.5.1
>
> Status in MadGraph5_aMC@NLO:
> New
>
> Bug description:
> Hello,
>
> In 2.5.1, we can run:
>
> $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC
>
> generate p p > t t~ [QCD]
> output
>
> where the installation directory is on a read-only file system, but
> `pwd` is read/write (e.g. a scratch area or tmp directory). This now
> asks about which programs to install, and then attempts two write a
> configuration file back to the read-only area:
>
> set golem ''
> save options --auto
> save configuration file to
> /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
> Command "output tt" interrupted with error:
> IOError : [Errno 30] Read-only file system:
> '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt'
>
> I believe this is because of a change in
>
> madgraph/interface/extended_cmd.py
>
> which now tries to save a configuration file via the
> write_configuration function when you try to do almost anything. I
> think, since we have the dependencies set to save locally, the correct
> behavior would be to make a locally directory and write there (or
> simply not update the config card at all).
>
> Thanks,
> Zach
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mg5amcnlo/+bug/1657615/+subscriptions

[Zachary Marshall (zach-marshall)]

Hi Olivier,

I don't love the idea of picking at random a process to run; I'd rather actually run the install commands if possible. Could you confirm that these are the right ones to cover most LO and NLO use cases?

install SysCalc Golem95 PJFry QCDLoop maddm collier ninja oneloop

I suppose the first point that you raised indicates that the default path in the config file points to a path that doesn't exist. Why wouldn't the default be set to none, and only if the user chooses to install one of these externals would the default config card be changed to use that location? If the user is installing an external then we know read/write is true, so it's safe to update the config card.

Thanks again,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

> I don't love the idea of picking at random a process to run;

Our code is actually designed at multiple place to configure itself at the time of the first process/loading of the code.
Such behaviour is actually especially useful when combine with the auto-update feature of MG5aMC (feature that you do not use).
This is perfectly fine in general but in the case of filesystem without write disk access since the code can not configure itself in that case.

This force you/us to find some work-around from each of the auto-configure step in order to have those working with your setup. (Copy-file, modifying configuration file,…)
I think that running one LO and one NLO process (you do not need to run them, just to “output” them) is actually the best way to handle such difficulty in the long term.
But this is obviously your call if you prefer to catch those problems one by one at each new release.

By the way, do you use the script compile.py? If yes, one correct solution would be that I run one LO and NLO generation from that script (and then clean the associate output obviously).

> Could you confirm that
> these are the right ones to cover most LO and NLO use cases?
> install SysCalc Golem95 PJFry QCDLoop maddm collier ninja oneloop

I do not know which of those library " Golem95 PJFry QCDLoop collier ninja oneloop” are compatible with your “set dependencies “ options.
Valentin will give more details on that.

Concerning SysCalc, we have now an internal python module handling the same functionalities (and more like you can change the dynamical scale definition).
Since the user support for SysCalc is not that great, I would suggest to use that one instead. It only requires to have lhapdf6 with python integration activated.

Additionally, Cuttools and iregi needs to be compile (we do not have any install for such tools which are automatically compiled when you “output” your first process with loop matrix-element.

> I suppose the first point that you raised indicates that the default
> path in the config file points to a path that doesn't exist.

Correct that path point to the default position where that program is expected to be installed.
In most of the case this is therefore not valid

> Why
> wouldn't the default be set to none, and only if the user chooses to
> install one of these externals would the default config card be changed
> to use that location?

Actually we use the fact that the “ninja” path is not valid to trigger the question on which reduction tools.
If the user specify that he did not want to install ninja then this path is set to None such that we do not ask the question again.

I can obviously prevent the asking of such question if I’m not in a write system. But in this context this does not sounds a good idea.

Cheers,

Olivier

> On 18 Feb 2017, at 04:40, Zachary Marshall <email address hidden> wrote:
>
> Hi Olivier,
>
> I don't love the idea of picking at random a process to run; I'd rather
> actually run the install commands if possible. Could you confirm that
> these are the right ones to cover most LO and NLO use cases?
>
> install SysCalc Golem95...

[Zachary Marshall (zach-marshall)]

Hi Olivier,

I recognize that most of your users are working on their own systems that they have full access to, but I do think our production workflow is a case that should be supported well.

I don't believe we're using the compile.py script at the moment, but I would be very happy to make that the standard for installation if you can integrate the additional steps into it. My point about running arbitrary processes is that it's not clear to me if a random process I pick (NLO ttbar?) will always be guaranteed to require all the externals that an arbitrary process will require. That's why I prefer being explicit about the list of externals. Of course, the ideal is that the compile.py script has either its own list, or a set of processes that installs the appropriate externals for an arbitrary process. So in a nutshell, yes, I'd be very happy with that update to compile.py!

Thanks,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi Zach,

So I have update the official compile.py script. You will find the patch in attachment.

So this code does the following:
- compile all the UFO model (not required but speed up the use of those model)
- compile all the v4 model (not required but speed up the use of those model)
- compile stdhep (required for NLO computation with hw6/py6 shower)
- compile Cuttools/Iregi (required for loop-computation)
- install a given list of external/optional program. The default list is related to our cluster (so I would not recommend that one in your case):
pythia-pgs, Delphes, ExRootAnalysis, MadAnalysis4, SysCalc
you can change the list by setting the list as argument of the script (e.g. ./bin/compile.py maddm SysCalc ninja)
- make a temporary output for a LO and NLO process to trigger all the auto-configure that exists.
- force the pre-compilation (in debug mode and normal mode) for all the python file.

If some of those steps does not match your need. We can add some option like "--no-stdhep" such that you can control what step you want to run.

Cheers,

Olivier

[Valentin Hirschi (valentin-hirschi)]

Hi Zach, Olivier,

On Sat, Feb 18, 2017 at 2:29 PM, Olivier Mattelaer <
<email address hidden>> wrote:

> Hi Zach,
>
> > I don't love the idea of picking at random a process to run;
>
> Our code is actually designed at multiple place to configure itself at the
> time of the first process/loading of the code.
> Such behaviour is actually especially useful when combine with the
> auto-update feature of MG5aMC (feature that you do not use).
> This is perfectly fine in general but in the case of filesystem without
> write disk access since the code can not configure itself in that case.
>
> This force you/us to find some work-around from each of the auto-configure
> step in order to have those working with your setup. (Copy-file, modifying
> configuration file,…)
> I think that running one LO and one NLO process (you do not need to run
> them, just to “output” them) is actually the best way to handle such
> difficulty in the long term.
> But this is obviously your call if you prefer to catch those problems one
> by one at each new release.
>
> By the way, do you use the script compile.py? If yes, one correct
> solution would be that I run one LO and NLO generation from that script
> (and then clean the associate output obviously).
>
> > Could you confirm that
> > these are the right ones to cover most LO and NLO use cases?
> > install SysCalc Golem95 PJFry QCDLoop maddm collier ninja oneloop
>
> I do not know which of those library " Golem95 PJFry QCDLoop collier ninja
> oneloop” are compatible with your “set dependencies “ options.
> Valentin will give more details on that.
>

Ok, so first of all the only dependencies that make sense to install for
production are:

CutTools, IREGI, Ninja and COLLIER

The first two (CutTools and IREGI) are not so useful but there mainly
because one cannot turned them off. This is a technical limitation that is
inherited from the history of the code development and that we should
eventually lift; but for now they are mandatory.
The option 'output_dependencies' basically only applies to them, and for
them two useful options are:

> internal: The whole source code of CutTools and IREGI is copied in the
output process directory and automatically recompiled when necessary when
launching an aMC@NLO run.

> external: The compiled fortran90 modules and static libraries are linked
'libcts.a libiregi.a mpmodule.mod' to the lib directory of the
process_output and therefore the target of the links must be available at
all time when launching/running the aMC@NLO code.

For all other loop-related dependencies, the option 'output_dependencies'
has no impact (or another way of saying it is: I have not implemented the
'internal' mode for them).
This means that the two additional relevant dependencies Ninja and COLLIER
(which are more powerful implementations of the CutTools and IREGI
counterparts), there is only two options of the user (assuming the other
three dependencies Golem95, PJFry and Samurai have been turned off in
mg5_configuration.txt, which I recommend you do anyway):

a) Disable them all together by specifying 'None' or '' for their
corresponding install path in mg5_configuration.txt. MadLoop will be able
to...

[Zachary Marshall (zach-marshall)]

Thanks Olivier and Valentin, this is very helpful! So far, it sounds like we should do something like:

./bin/compile.py SysCalc ninja CutTools IREGI collier

I just want to check that I understand two things Valentin said. You recommend that we do *not* install Golem95, PJFry, and Samurai, correct? And if we install the dependencies above will we still need to modify the mg5_configuration.txt file by hand before running, or should this now work properly out of the box?

Thanks again,
Zach

[Valentin Hirschi (valentin-hirschi)]

On Mon, Feb 20, 2017 at 8:18 PM, Zachary Marshall <email address hidden>
wrote:

> Thanks Olivier and Valentin, this is very helpful! So far, it sounds
> like we should do something like:
>
> ./bin/compile.py SysCalc ninja CutTools IREGI collier
>
> I just want to check that I understand two things Valentin said. You
> recommend that we do *not* install Golem95, PJFry, and Samurai, correct?
>

Correct should you definitely not have to install Golem95, PJFry or
Samurai.
Also, the sub-dependency OneLOop will be handled automatically behind the
scene, so you shouldn't have to worry about it.

> And if we install the dependencies above will we still need to modify
> the mg5_configuration.txt file by hand before running, or should this
> now work properly out of the box?
>

Now, the mg5_configuration.txt should be automatically set accordingly to
the MG5_aMC steered installation done by compile.py (please Olivier
confirm).

You will only need to fiddle with the soft links or paths in the process
output directory *if and only if* you plan on distributing the output
process on machines that do not have access to the tools installed during
the compile.py scripts (all of them are placed in a HEPTools directory that
should ideally be accessible from all machines that intend to use this
MG5_aMC distribution).

Let us know if you need further information to make your centralized
MG5_aMC installation work.

>
> Thanks again,
> Zach
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1657615
>
> Title:
> Configuration writing to a read-only file system in 2.5.1
>
> Status in MadGraph5_aMC@NLO:
> Fix Committed
>
> Bug description:
> Hello,
>
> In 2.5.1, we can run:
>
> $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC
>
> generate p p > t t~ [QCD]
> output
>
> where the installation directory is on a read-only file system, but
> `pwd` is read/write (e.g. a scratch area or tmp directory). This now
> asks about which programs to install, and then attempts two write a
> configuration file back to the read-only area:
>
> set golem ''
> save options --auto
> save configuration file to
> /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-
> opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/
> madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
> Command "output tt" interrupted with error:
> IOError : [Errno 30] Read-only file system:
> '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-
> opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/
> madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
> '
>
> I believe this is because of a change in
>
> madgraph/interface/extended_cmd.py
>
> which now tries to save a configuration file via the
> write_configuration function when you try to do almost anything. I
> think, since we have the dependencies set to save locally, the correct
> behavior would be to make a locally directory and write there (or
> simply not update the config card at all).
>
> Thanks,
> Zach
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mg5amcnlo/+bug/165761...

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

The correct command is a

>> ./bin/compile.py SysCalc ninja collier

Or

>> ./bin/compile.py SysCalc ninja collier maddm

CutTools and Iregi will be install systematically.

> Now, the mg5_configuration.txt should be automatically set accordingly to
> the MG5_aMC steered installation done by compile.py (please Olivier
> confirm).

Yes it should be.

Cheers,

Olivier

> On 20 Feb 2017, at 20:48, Valentin Hirschi <email address hidden> wrote:
>
> On Mon, Feb 20, 2017 at 8:18 PM, Zachary Marshall <email address hidden>
> wrote:
>
>> Thanks Olivier and Valentin, this is very helpful! So far, it sounds
>> like we should do something like:
>>
>> ./bin/compile.py SysCalc ninja CutTools IREGI collier
>>
>> I just want to check that I understand two things Valentin said. You
>> recommend that we do *not* install Golem95, PJFry, and Samurai, correct?
>>
>
> Correct should you definitely not have to install Golem95, PJFry or
> Samurai.
> Also, the sub-dependency OneLOop will be handled automatically behind the
> scene, so you shouldn't have to worry about it.
>
>
>> And if we install the dependencies above will we still need to modify
>> the mg5_configuration.txt file by hand before running, or should this
>> now work properly out of the box?
>>
>
> Now, the mg5_configuration.txt should be automatically set accordingly to
> the MG5_aMC steered installation done by compile.py (please Olivier
> confirm).
>
> You will only need to fiddle with the soft links or paths in the process
> output directory *if and only if* you plan on distributing the output
> process on machines that do not have access to the tools installed during
> the compile.py scripts (all of them are placed in a HEPTools directory that
> should ideally be accessible from all machines that intend to use this
> MG5_aMC distribution).
>
> Let us know if you need further information to make your centralized
> MG5_aMC installation work.
>
>
>>
>> Thanks again,
>> Zach
>>
>> --
>> You received this bug notification because you are a bug assignee.
>> https://bugs.launchpad.net/bugs/1657615
>>
>> Title:
>> Configuration writing to a read-only file system in 2.5.1
>>
>> Status in MadGraph5_aMC@NLO:
>> Fix Committed
>>
>> Bug description:
>> Hello,
>>
>> In 2.5.1, we can run:
>>
>> $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC
>>
>> generate p p > t t~ [QCD]
>> output
>>
>> where the installation directory is on a read-only file system, but
>> `pwd` is read/write (e.g. a scratch area or tmp directory). This now
>> asks about which programs to install, and then attempts two write a
>> configuration file back to the read-only area:
>>
>> set golem ''
>> save options --auto
>> save configuration file to
>> /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-
>> opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/
>> madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
>> Command "output tt" interrupted with error:
>> IOError : [Errno 30] Read-only file system:
>> '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-
>> opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/
>> madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/in...

[Ivan Razumov (iarspider)]

Hi Olivier, Valentin,

I'm currently the person maintaining the centralized installation that Zachary is using. The tarball of MadGraph 2.5.2 that is available on LP (https://launchpad.net/mg5amcnlo/2.0/2.5.x/+download/MG5_aMC_v2.5.2.tar.gz) does not have compile.py script - is it safe to take one from the repository?

Best regards,
Ivan for the GENSER team

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Yes using the one from the repository is ok.
If you want you can download the patched version from the following link:
http://bazaar.launchpad.net/~maddevelopers/mg5amcnlo/2.5.3/download/head:/compile.py-20110413144801-gz8gzwj02idw713s-1/compile.py

Cheers,

Olivier

PS: If you want us to include compile.py in the tar ball this is obviously not a problem.

http://bazaar.launchpad.net/~maddevelopers/mg5amcnlo/2.5.3/download/head:/compile.py-20110413144801-gz8gzwj02idw713s-1/compile.pyOn 22 Feb 2017, at 13:29, Ivan Razumov <<email address hidden><mailto:<email address hidden>>> wrote:

Hi Olivier, Valentin,

I'm currently the person maintaining the centralized installation that
Zachary is using. The tarball of MadGraph 2.5.2 that is available on LP
(https://launchpad.net/mg5amcnlo/2.0/2.5.x/+download/MG5_aMC_v2.5.2.tar.gz)
does not have compile.py script - is it safe to take one from the
repository?

Best regards,
Ivan for the GENSER team

--
You received this bug notification because you are subscribed to
MadGraph5_aMC@NLO.
https://bugs.launchpad.net/bugs/1657615

Title:
 Configuration writing to a read-only file system in 2.5.1

Status in MadGraph5_aMC@NLO:
 Fix Committed

Bug description:
 Hello,

 In 2.5.1, we can run:

 $MADGRAPH_INSTALLATION_DIR/bin/mg5_aMC

 generate p p > t t~ [QCD]
 output

 where the installation directory is on a read-only file system, but
 `pwd` is read/write (e.g. a scratch area or tmp directory). This now
 asks about which programs to install, and then attempts two write a
 configuration file back to the read-only area:

 set golem ''
 save options --auto
 save configuration file to
 /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt
 Command "output tt" interrupted with error:
 IOError : [Errno 30] Read-only file system:
 '/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.1.atlas/x86_64-slc6-gcc47-opt/input/mg5_configuration.txt'

 I believe this is because of a change in

 madgraph/interface/extended_cmd.py

 which now tries to save a configuration file via the
 write_configuration function when you try to do almost anything. I
 think, since we have the dependencies set to save locally, the correct
 behavior would be to make a locally directory and write there (or
 simply not update the config card at all).

 Thanks,
 Zach

To manage notifications about this bug go to:
https://bugs.launchpad.net/mg5amcnlo/+bug/1657615/+subscriptions

[Zachary Marshall (zach-marshall)]

Hi,

I think it would be very helpful to include the compile.py script in the tarball (and, happily, it should be pretty small).

Best,
Zach

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

Ok I have added it but as a hidden file (./bin/.compile.py)
Such that it will not confuse standard user who should not use such program (especially since by default it installed outdated library like pythia-pgs)

Cheers,

Olivier

[Zachary Marshall (zach-marshall)]

Hi guys,

Sorry, it looks like we aren't quite there. I thought that this would now pick up externals from the installation area. When I run:

generate p p > t t~ [QCD]
output -f

and then later:

./bin/generate_events --name=testRun

I see:

INFO: Compiling source...
INFO: ...done, continuing with P* directories
INFO: Compiling StdHEP (can take a couple of minutes) ...
INFO: ...done.
INFO: Compiling IREGI (can take a couple of minutes) ...
...
            f77: cmatrix_base.f90: linker input file unused because linking not done
            f77 -c oneloop/avh_olo_foriregi.f90 -O -g -fPIC -I/tmp/zmarshal/test0/PROCNLO_loop_sm_0/Source/IREGI/src/oneloop
            f77: oneloop/avh_olo_foriregi.f90: linker input file unused because linking not done
            make[2]: *** No rule to make target `avh_olo_foriregi_dp_box.mod', needed by `mis_warp.o'. Stop.
            make[2]: Leaving directory `/tmp/zmarshal/test0/PROCNLO_loop_sm_0/Source/IREGI/src'
            make[1]: *** [all] Error 2
            make[1]: Leaving directory `/tmp/zmarshal/test0/PROCNLO_loop_sm_0/Source/IREGI/src'
            make: *** [../lib/libiregi.a] Error 2

The externals are installed as we discussed:

$ ls /cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc6-gcc47-opt/19.2.5/sw/lcg/external/MCGenerators_lcgcmt67c/madgraph5amc/2.5.2.atlas/x86_64-slc6-gcc47-opt/vendor/
CutTools DiscreteSampler IREGI OfflineHEPToolsInstaller.tar.gz SMWidth StdHEP __init__.py collier.tar.gz ninja.tar.gz oneloop.tar.gz ply

But they don't seem to be picked up. I see that they're meant to be searched for in an auto-generated path, but I'm having trouble following where the path is really coming from. I also don't see any option in the config card to specify the location of IREGI or these other externals. What am I missing here?

Thanks,
Zach

[Valentin Hirschi (valentin-hirschi)]

Ok, this issue was already unravelled and fixed in 2.5.3 (see rev. http://bazaar.launchpad.net/~olivier-mattelaer/mg5amcnlo/2.5.3/revision/306).

Please let us know if it were still to be a problem in 2.5.3.