Buggy data in capillary files

[Jérôme Duriez]

Hi,

***** CURRENT SITUATION ******
Current capillary files include some buggy data for dimensionless menisci volume (vol*), and filling angle delta2. Several examples are detailled in "prob.txt" attachment.

 Basically, on some lines, vol* and delta2 are discontinuous compared to adjacent lines. The discontinuous values are sometimes obviously wrong, with e.g. one vol* = 400 line 9704 of M(r=2). You may imagine the consequences on the construction of a soil-water characteristic curve...
*******************************

****** HISTORY (explaining the current situation ?) ******
Historically, these files come from a very first version of capillary files, that have been corrected twice. A first ("major") correction modified the volume values that were affected by a mistake during files construction (*). There has been a second ("minor") correction regarding the first lines of some files (**). Probably the discontinuous volume values existing now have been introduced during the major correction (*).
******************************************************

****** CORRECTION PROPOSAL ********
After discussion with Luc Scholtes, some work has been done to correct the buggy volume / delta2 values that exist now. It has been decided to start from the very first version of the files. In this very first version no discontinuous volume values existed.

But, discontinuous values of delta2, sometimes reaching - 6000 degrees have been now spotted in these very first files.
To correct this, the lines including such discontinuous delta2 values have been simply erased. To do so, continuity has been checked with adjacent lines, spotting then errors like delta2 = - 6000 deg between delta2 = 55 and 54 degrees, and also delta2 = 3 degrees between 22 and 22.
Doing so, roughly less than 1% of lines have been erased.

Discontinuous lines being erased, volume values have been corrected towards the same direction than during correction (*).
**************************************

****** FINAL SITUATION AFTER POSSIBLE APPROVAL OF PROPOSED CORRECTION*******
This work has been completed. It is not uploaded yet, time being given for people to check the current situation, and possible discussion about the correction proposal.

In the brand new files (only in my PC as for now), there should be no discontinuous values of vol* and delta2 anymore, and volume meniscii should be correct.
***********************************************************************************

(*) https://answers.launchpad.net/yade/+question/201608
(**) http://www.mail-archive.com/yade-dev%40lists.launchpad.net/msg10638.html

[Václav Šmilauer (eudoxos)]

Hi Jerome, is the code to generate those files in the github repo yet? Cheers, v.

[Jérôme Duriez (jduriez)]

Hi Vaclav,

No, and I do not have this code myself.
What I have is a MATLAB code I used to modify the very first version of the capillary files (I'm attaching). Luc Scholtes provided me these very first version of the files.
I guess he is the only one, if any, who may have the code to generate from scratch these capillary files.

(However, I was thinking it could be useful to include these text files in "trunk" to ease control, any thoughts ?)

[Bruno Chareyre (bruno-chareyre)]

Hi,
As Vaclav suggests the real solution is to provide a matlab/scilab/python/whichever script generating the input data.
I can provide a scilab script doing the math - not a big deal, but there is a little bit of work left to write the results in the correct format. If someone is willing to try that let me know.
If not possible, ok to commit improved files.

Bruno

p.s. I wonder why those junk values are not visible in most cases, for instance when plotting the evolution of volume or force vs. distance, any idea?

[Jérôme Duriez (jduriez)]

I do not know if I could take care of re-generating new files, but I am personly anyway curious about your script, if you agree to provide it (would also help to assess the required work amount).

Fresh new files being generated or not, we agree to put now capillary files inside sources (in an "extra" subfolder ?) instead on wiki, that's it ?

[Bruno Chareyre (bruno-chareyre)]

>we agree to put now capillary files inside sources (in an "extra" subfolder ?)

The data files are too big. Not in source. Let's keep only source code in source code.
I'm sure there is a python or c++ command to download them when needed.
B

[Jérôme Duriez (jduriez)]

Ok. Last question/remark before I upload the corrected files :

in M(r=10), there is a bunch of "missing" uc* values in some cases. For dist* = 0 and 1e-4, considered uc* values jump from 1350 to 11000. This being said, trends in all values changes (according to uc*) do not show anything chocking, in this file.

About your p.s. question of #7, the junk values seem to correspond to a set of few given uc* values. As such, as soon as you consider an example with a given capillary pressure that do not fit those uc* values, nothing appears.
Note also that I did not spot any problem about force values.
(If I understood your question)

[Bruno Chareyre (bruno-chareyre)]

Yes you understood my question, thanks.
The thing is plotting force=force(uc*) is not uncommon. It goes through all values then it should show weird evolutions somewhere, but nothing.
I wonder if the junk values are not just sorted out automaticaly (by chance) by the interpolation method.
B

[Jérôme Duriez (jduriez)]

Ok, so my opinion is that there is no junk force values in the files. Then, force(uc*) curves have nothing to reveal.

However, there are junk volume / delta2 values (for some dist*), that get obvious on volume(uc*), or delta2(uc*), curves. See "Figure ..." bug attachments

In the end, I think personally that the lines that include junk volume / delta2 values (but not any junk force values) do not obey Laplace's equation. For some reason...

[Jérôme Duriez (jduriez)]

I just uploaded the corrected files on the wiki.

I spotted finally a small problem in M(r=10), that might be considered as negligible (see attachment). However, for reasons explained in #10, this file is the only one unchanged.

[Jérôme Duriez (jduriez)]

Hi,

I'm finally working on open source scripts to regenerate capillary files from scratch. Hopefully, there will be a commit in a couple of months.
In "Luc Scholtes' capillary files", the d* values were not linearly spaced, as the attached figure shows it (it shows the increment between two successive d* values, and it is not constant)

I see the point not taking an constant increment, but, at the - far ... - time these files were generated, was there a precise reason to take such non linear d* distribution, rather than another (non linear) ?

Thanks,

Jerome

PS : the attached figure applies to d* values as in the capillary files before June 2012, and since April 2015. Between these dates, the "d* structure" was a bit different, with e.g. roughly 27 d* values instead of 50-55

[Jérôme Duriez (jduriez)]

PPS: the same question holds more or less for the uc* values

[Jérôme Duriez (jduriez)]

The current uc* values distribution, for the record

[Jérôme Duriez (jduriez)]

Ok, it is in fact quite clear for what concerns uc*

(sorry for the possible spam, the record is sometimes also for me)

[Bruno Chareyre (bruno-chareyre)]

Hi Jerome,
It would be a good thing to have scripts generating data in trunk indeed.
"d" is a solution of the ODE integration, not an input variable, that is why you don't find a logic in the increments.
"uc" is an input parameter OTOH. I don't remember in detail how its evolution was defined by I don't see why exactly a linear evolution would be better

>with e.g. roughly 27 d* values instead of 50-55

It sounds like a problem. What determines accuracy is primarily the local density of data points, no matter how the values are incremented. So 27 values will give less accurate interpolation than 50 values overall.

[Jérôme Duriez (jduriez)]

To be sure I got you (and you got me):

In the files, this distance does have a well defined set of values: with e.g. (I'm taking random figures) 600 lines at d = cst = 0, then 500 lines at d = cst = 1e-4, then 400 lines at d = cst = 5e-3.

Up to now, I would not be able to obtain such precise set of values directly from solving 600 + 500 + 400 times the ODE. That's why I think there is a second step, after the ODE solving, to construct the capillary files. During which a choice for the d values has to be made. Don't you think so ?

[Bruno Chareyre (bruno-chareyre)]

This is a suprise for me then, I was not expecting fixed values of d.
If Luc had a trick to achieve this only him can explain now, I forgot. It is not necessary to have this feature in general, but it could be that the interpolation algorithm requires that (with Caroline's code it does not).

[Jérôme Duriez (jduriez)]

Yes, the interpolation algorithm does require it.
See https://github.com/yade/trunk/blob/master/pkg/dem/Law2_ScGeom_CapillaryPhys_Capillarity.hpp#L108 and all "full_data[XX].D" in the cpp.
I should be able to manage it, thank you !