Lack of specifications for test cases

[Yann Pouillon]

I'm currently carrying out a testing campaign on various builds of the trunk of SIESTA (revno 646). I want to automate everything as much as possible, in order to avoid human mistakes and schedule the tests when the computer is most available.

In most cases, I can have a generic shell-script calling 'make' for individual tests. However, I'm not able to decide what to do for some test cases or have to manually change the script to run it successfully. I will use a few examples here to make questions about the design of the test suite.

Example 1: TDDFT_Ge contains 2 input files, TTDFT_Ge1.fdf and TDDFT_Ge2.fdf. Which one should be run? Does the order matter?

Example 2: The input files of the flos_h2o_* tests are called h2o_*.fdf, which means I cannot automatically check their presence by using the directory name of the test. Is this voluntary?

Example 3: The TRANSIESTA tests have the same issue as example 2, in addition to be in subdirectories of TranSiesta-TBTrans/. I can deal with directory hierarchies quite easily, but their combination with changes in input file names makes it a bit cumbersome.

I would like to discuss design principles for test cases, in order to determine if we can converge on a minimum set of simple criteria when writing new tests. This will be particularly useful to me when adding the tests for the hybrids.

[Nick Papior (nickpapior)]

Example 1.
When running make in the TDDFT_Ge sub-directory it will run _1 first, then _2. This should be controlled by the test_tddft.mk file. However, I agree it could be updated for clarity.

Example 2.
I will go through the tests in 4.0 and update names to be consistent.

Example 3.
Currently this is limited because one has to compile transiesta as a separate executable. Hence those tests are sort of different. Probably this would change if I get the time to make transiesta a part of siesta (so one does not need to different executables).
In that case then we could prefix transiesta tests with TS_, just like TDDFT.

[Yann Pouillon (pouillon)]

Thanks. Clarity is definitely what is needed. A few simple recommendations on how to design test cases would likely be enough.

[Nick Papior (nickpapior)]

Example 2 has just been fixed in 4.1, this will propagate to trunk (will let you know). Once merged you can pull into your branch.

[Nick Papior (nickpapior)]

Feel free to merge with trunk now.