Lack of specifications for test cases
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Siesta |
New
|
Undecided
|
Unassigned |
Bug Description
I'm currently carrying out a testing campaign on various builds of the trunk of SIESTA (revno 646). I want to automate everything as much as possible, in order to avoid human mistakes and schedule the tests when the computer is most available.
In most cases, I can have a generic shell-script calling 'make' for individual tests. However, I'm not able to decide what to do for some test cases or have to manually change the script to run it successfully. I will use a few examples here to make questions about the design of the test suite.
Example 1: TDDFT_Ge contains 2 input files, TTDFT_Ge1.fdf and TDDFT_Ge2.fdf. Which one should be run? Does the order matter?
Example 2: The input files of the flos_h2o_* tests are called h2o_*.fdf, which means I cannot automatically check their presence by using the directory name of the test. Is this voluntary?
Example 3: The TRANSIESTA tests have the same issue as example 2, in addition to be in subdirectories of TranSiesta-
I would like to discuss design principles for test cases, in order to determine if we can converge on a minimum set of simple criteria when writing new tests. This will be particularly useful to me when adding the tests for the hybrids.
Example 1.
When running make in the TDDFT_Ge sub-directory it will run _1 first, then _2. This should be controlled by the test_tddft.mk file. However, I agree it could be updated for clarity.
Example 2.
I will go through the tests in 4.0 and update names to be consistent.
Example 3.
Currently this is limited because one has to compile transiesta as a separate executable. Hence those tests are sort of different. Probably this would change if I get the time to make transiesta a part of siesta (so one does not need to different executables).
In that case then we could prefix transiesta tests with TS_, just like TDDFT.