Opening qt.Controller() might truncate O.dt value

[Václav Šmilauer]

First reported by Anton on the ML. The attached script produces different results for same initial simulation. By running it, it will produce 3 saved final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are different. It should be found out why.

[Bruno Chareyre (bruno-chareyre)]

I realized there is some randomness in this simulation as well.
The results are similar if the simulation is started exactly the same way,
but if I run "valgrind --tool=callgrind ./yade file" instead of "./yade file" the results are different after 15k iterations.

I suspsect a very small difference somewhere is causing a significant (but not dramatic) difference after many iterations, as in classical chaotic systems.

You can compare the results with gnuplot : "plot 'Wallstresses' using 0:2", different simulations are appended to the same file.

Is there something related with cpu clock somewhere in Yade?

[Václav Šmilauer (eudoxos)]

The periodic engines can be clock-related in general, but in this case there is

 PeriodicPythonRunner(iterPeriod=1000,label='plotDataCollector',command='myAddPlotData()')

meaning the engine runs every 1000 steps, which doesn't depend on time (if it did set realPeriod, only then it would depend on wall clock time).

Otherwise I am not aware of anything clock-related. If the initial states are really identical (loaded from the same .xml file), there should not be even the "small" differences.

Although, it reminds me, the gas molecular dynamics guy from Germany was complaining at the conference in Aussois last year that variables can be slightly differently rounded depending on whether they are kept in register (80bits) or moved to memory (64bit?) before being used again for computation. And it is up to hte CPU logic to decide which cache lines to flush at which point, and depends on other processes etc as well.

But I don't think that this is the problem here. The curves look really too much different.

I will try to run two simulations at the same time (using O.switchWorld()) step by step and comparing them at every step, to see where the difference comes from. I will post that script if it works somehow.

[Václav Šmilauer (eudoxos)]

Can you try the attached script with r1845 (it runs the thing 3 times, saves *.gnuplot files)? I didn't change anything that (as far as I know) is related, but I am not able to reproduce the behavior anymore. The results are exactly the same for OMP_NUM_THREADS=1 and they differ only slightly for multiple threads. If it works, you can close this bug.

It is still suspicious for the multi-threaded a little bit:

126000 0.0805152159531 0.0867205790231 -0.0234114094128
127000 0.0811542256035 0.0805748256519 -0.0234190939172

126000 0.0805152159531 0.0867205790231 -0.0234114094128
127000 0.0811542256035 0.0807008266197 -0.0234190539184

The first lines are still identical, but 0.08672 vs. 0.0807008 (3rd column) is quite some difference!

[Václav Šmilauer (eudoxos)]

Ah, got the cause, it seems: Cundall's damping. It is a function with _discontinuity_ (i.e. not just with big but finite derivative), which means that it can turn instantaneously very small difference into a huge one. When running without damping (0.0), the results are 100% the same even for the multi-threaded run. Can you verify that?

[Anton Gladky (gladky-anton)]

Yes, of course!

But I can do it just tomorrow morning.
When I get results - will notify you.

Thank you!
______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/7 Václav Šmilauer <email address hidden>

> Ah, got the cause, it seems: Cundall's damping. It is a function with
> _discontinuity_ (i.e. not just with big but finite derivative), which
> means that it can turn instantaneously very small difference into a huge
> one. When running without damping (0.0), the results are 100% the same
> even for the multi-threaded run. Can you verify that?
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Confirmed
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Anton Gladky (gladky-anton)]

I cant run SVN 1845.
I get:
...
from miniWm3Wrap import *
ImportError: No module named miniWm3Wrap
...

______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/7 Anton Gladky <email address hidden>

> Yes, of course!
>
> But I can do it just tomorrow morning.
> When I get results - will notify you.
>
> Thank you!
> ______________________________
> [ENG] Best Regards
> [GER] Mit freundlichen Grüßen
> [RUS] С наилучшими пожеланиями
> [UKR] З найкращими побажаннями
>
> Anton Gladkyy
>
>
> 2009/7/7 Václav Šmilauer <email address hidden>
>
> Ah, got the cause, it seems: Cundall's damping. It is a function with
>> _discontinuity_ (i.e. not just with big but finite derivative), which
>> means that it can turn instantaneously very small difference into a huge
>> one. When running without damping (0.0), the results are 100% the same
>> even for the multi-threaded run. Can you verify that?
>>
>> --
>> different results for same initial simulation (uninitialized values?)
>> https://bugs.launchpad.net/bugs/394687
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>> Status in Yet Another Dynamic Engine: Confirmed
>>
>> Bug description:
>> First reported by Anton on the ML. The attached script produces different
>> results for same initial simulation. By running it, it will produce 3 saved
>> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
>> different. It should be found out why.
>>
>
>

[Václav Šmilauer (eudoxos)]

Did you recompile? This module should be built and installed automatically; depending on your PREFIX, it will be something like /usr/local/lib/yade-trunk-opt/py/miniWm3Wrap.so

[Anton Gladky (gladky-anton)]

Yes, I have recompiled it. Actually it makes automatically every night.
I can`t find this .so file. Only .cpp in trunk directory.
______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Václav Šmilauer <email address hidden>

> Did you recompile? This module should be built and installed
> automatically; depending on your PREFIX, it will be something like
> /usr/local/lib/yade-trunk-opt/py/miniWm3Wrap.so
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Confirmed
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Anton Gladky (gladky-anton)]

I have done 6 diagrams "by hand" (but thank you for good example, how to create several tests in one script)
It seems, you are right about NewtonsDampedLaw:
  when it is more, than 0 results are different.
  when it is 0 - results are equal.

[Václav Šmilauer (eudoxos)]

Huh, it was a file that I forgot, sorry! grab r1846

[Sergei Dorofeenko (sergei.dorofeenko)]

Václav Šmilauer пишет:
> Huh, it was a file that I forgot, sorry! grab r1846
>
>
py/miniWm3Wrap/miniWm3Wrap.cpp:9:75: error:
/home/vaclav/yade/trunk/py/miniWm3Wrap/miniWm3Wrap-toExpose.hpp: No such
file or directory

[Václav Šmilauer (eudoxos)]

r1847. Sorry. py++ puts absolute path automatically, I didn't spot that.

[Sergei Dorofeenko (sergei.dorofeenko)]

Hm... I have identical results for different num threads and dampings.

[Václav Šmilauer (eudoxos)]

My results also looked identical with multiple threads, but the numbers differed at 3rd decimal place, for instance, which is not visible in the graph, but you see it if you compare numbers in *.data.bz2 files.

[Anton Gladky (gladky-anton)]

Try to use more iterations. For example 200 000.

______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Václav Šmilauer <email address hidden>

> My results also looked identical with multiple threads, but the numbers
> differed at 3rd decimal place, for instance, which is not visible in the
> graph, but you see it if you compare numbers in *.data.bz2 files.
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Confirmed
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Anton Gladky (gladky-anton)]

Actually, I got a "good" difference on 500 000 iterations
______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Anton Gladky <email address hidden>

> Try to use more iterations. For example 200 000.
>
> ______________________________
> [ENG] Best Regards
> [GER] Mit freundlichen Grüßen
> [RUS] С наилучшими пожеланиями
> [UKR] З найкращими побажаннями
>
> Anton Gladkyy
>
>
> 2009/7/8 Václav Šmilauer <email address hidden>
>
>> My results also looked identical with multiple threads, but the numbers
>>
>> differed at 3rd decimal place, for instance, which is not visible in the
>> graph, but you see it if you compare numbers in *.data.bz2 files.
>>
>> --
>> different results for same initial simulation (uninitialized values?)
>> https://bugs.launchpad.net/bugs/394687
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>> Status in Yet Another Dynamic Engine: Confirmed
>>
>> Bug description:
>> First reported by Anton on the ML. The attached script produces different
>> results for same initial simulation. By running it, it will produce 3 saved
>> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
>> different. It should be found out why.
>>
>
>

[Anton Gladky (gladky-anton)]

Vaclav, I've got files results with different results.
But, I dont know how to save gnuplot images to file.
I've attached file.

______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Anton Gladky <email address hidden>

> Actually, I got a "good" difference on 500 000 iterations
> ______________________________
> [ENG] Best Regards
> [GER] Mit freundlichen Grüßen
> [RUS] С наилучшими пожеланиями
> [UKR] З найкращими побажаннями
>
> Anton Gladkyy
>
>
> 2009/7/8 Anton Gladky <email address hidden>
>
> Try to use more iterations. For example 200 000.
>>
>> ______________________________
>> [ENG] Best Regards
>> [GER] Mit freundlichen Grüßen
>> [RUS] С наилучшими пожеланиями
>> [UKR] З найкращими побажаннями
>>
>> Anton Gladkyy
>>
>>
>> 2009/7/8 Václav Šmilauer <email address hidden>
>>
>>> My results also looked identical with multiple threads, but the numbers
>>>
>>> differed at 3rd decimal place, for instance, which is not visible in the
>>> graph, but you see it if you compare numbers in *.data.bz2 files.
>>>
>>> --
>>> different results for same initial simulation (uninitialized values?)
>>> https://bugs.launchpad.net/bugs/394687
>>> You received this bug notification because you are a direct subscriber
>>> of the bug.
>>>
>>> Status in Yet Another Dynamic Engine: Confirmed
>>>
>>> Bug description:
>>> First reported by Anton on the ML. The attached script produces different
>>> results for same initial simulation. By running it, it will produce 3 saved
>>> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
>>> different. It should be found out why.
>>>
>>
>>
>

[Václav Šmilauer (eudoxos)]

The "00" looks same to me, the "05" has results quite different, but what params did you use?

[Anton Gladky (gladky-anton)]

NewtonsDampedLaw(damping=0.5)

______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Václav Šmilauer <email address hidden>

> The "00" looks same to me, the "05" has results quite different, but
> what params did you use?
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Confirmed
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Anton Gladky (gladky-anton)]

It is 500 000 iterations
______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/7/8 Anton Gladky <email address hidden>

> NewtonsDampedLaw(damping=0.5)
>
> ______________________________
> [ENG] Best Regards
> [GER] Mit freundlichen Grüßen
> [RUS] С наилучшими пожеланиями
> [UKR] З найкращими побажаннями
>
> Anton Gladkyy
>
>
> 2009/7/8 Václav Šmilauer <email address hidden>
>
>> The "00" looks same to me, the "05" has results quite different, but
>>
>> what params did you use?
>>
>> --
>> different results for same initial simulation (uninitialized values?)
>> https://bugs.launchpad.net/bugs/394687
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>> Status in Yet Another Dynamic Engine: Confirmed
>>
>> Bug description:
>> First reported by Anton on the ML. The attached script produces different
>> results for same initial simulation. By running it, it will produce 3 saved
>> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
>> different. It should be found out why.
>>
>
>

[Václav Šmilauer (eudoxos)]

We already know that damping may make the results different. What is the problem?

[Václav Šmilauer (eudoxos)]

Seems clear now. Please reopen if you still have issues with different results without damping and with OMP_NUM_THREADS==1. With damping and with multi-threaded operation, results are allowed to differ.

[Anton Gladky (gladky-anton)]

http://www.thinkingparallel.com/2006/08/21/scoped-locking-vs-critical-in-openmp-a-personal-shootout/
This was the reason?

______________________________

Anton Gladkyy

2009/7/15 Václav Šmilauer <email address hidden>

> Seems clear now. Please reopen if you still have issues with different
> results without damping and with OMP_NUM_THREADS==1. With damping and
> with multi-threaded operation, results are allowed to differ.
>
> ** Changed in: yade
> Status: Confirmed => Fix Released
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Fix Released
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Václav Šmilauer (eudoxos)]

No, that has mostly nothing to do with that. Just that everybody knows now that slightly different results are OK with openMP (different addition order, hence different rounding; if damping!=0, that can make very different results) and it is not a bug.

[Anton Gladky (gladky-anton)]

Hmm, still have it. Even with OMP_NUM_THREADS=1

[Václav Šmilauer (eudoxos)]

Is the differences you have related to distanceFactor as we talked about on chat? If so, I would open a new bug, since this was about divergence in general.

You can try scripts/compare-identical.py. Save your simulation into .xml, adjust initFile at the beginning of the script and run it. It is a pure xml, though, so you cannot call python functions that were defined in your .py file only.

(You could change that: define functions separately, then instead of O.load(initFile) in the loop over 2 "worlds", do the simulation setup itself, as normally. Note that plot results will clash one with another, but that shouldn't be a big deal)

[Anton Gladky (gladky-anton)]

Ok, will try to do it.
______________________________
[ENG] Best Regards
[GER] Mit freundlichen Grüßen
[RUS] С наилучшими пожеланиями
[UKR] З найкращими побажаннями

Anton Gladkyy

2009/8/5 Václav Šmilauer <email address hidden>

> Is the differences you have related to distanceFactor as we talked about
> on chat? If so, I would open a new bug, since this was about divergence
> in general.
>
> You can try scripts/compare-identical.py. Save your simulation into
> .xml, adjust initFile at the beginning of the script and run it. It is a
> pure xml, though, so you cannot call python functions that were defined
> in your .py file only.
>
> (You could change that: define functions separately, then instead of
> O.load(initFile) in the loop over 2 "worlds", do the simulation setup
> itself, as normally. Note that plot results will clash one with another,
> but that shouldn't be a big deal)
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Fix Released
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>

[Anton Gladky (gladky-anton)]

Vaclav, sorry to write in old bug, but it seems, that the problem is in
DISPLAY= parameter.
I still had problems with different results with different number of
threads. I used just OMP_NUM_THREADS=1 to get identical results.

But when I deleted DISPLAY= parameter from multy-starter, I have no any
problems with number of threads. I can start the same simulations with any
threads and results are identical the difference is only in calculation
speed, what is obvious.

Maybe I am wrong, but it works for me.

Thank you for your help.

2009/7/15 Václav Šmilauer <email address hidden>

> .......Just that everybody knows
> now that slightly different results are OK with openMP (different
> addition order, hence different rounding; if damping!=0, that can make
> very different results) and it is not a bug.
>
>

[Václav Šmilauer (eudoxos)]

I am reopening this bug; can yuou (Anton) provide some testcase for this one? Don't you have something like

try:

[Václav Šmilauer (eudoxos)]

I am reopening this bug; can yuou (Anton) provide some testcase for this one? Don't you have something like

try:
   from yade import qt
   qt.Controller()
   someParameter=34534
except ImportError:
   pass

?

That would explain the difference, as if DISPLAY is empty or not set, yade.qt raises ImportError. That would skip the assignment, then you'd get different results.

Can you save the simulation right at the start and compare both?

I can't have a look in depth myself now.

[Anton Gladky (gladky-anton)]

Yes, I have it in my simulation.

You are right, first 2 steps in my simulation are before those strings and
results are the same.
The difference appears after "try: from yade import qt......"

I will send test script as quick as possible.

Thank you.

2010/4/14 Václav Šmilauer <email address hidden>

> I am reopening this bug; can yuou (Anton) provide some testcase for this
> one? Don't you have something like
>
> try:
> from yade import qt
> qt.Controller()
> someParameter=34534
> except ImportError:
> pass
>
> ?
>
> That would explain the difference, as if DISPLAY is empty or not set,
> yade.qt raises ImportError. That would skip the assignment, then you'd
> get different results.
>
> Can you save the simulation right at the start and compare both?
>
> I can't have a look in depth myself now.
>
> --
> different results for same initial simulation (uninitialized values?)
> https://bugs.launchpad.net/bugs/394687
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Yet Another Dynamic Engine: Incomplete
>
> Bug description:
> First reported by Anton on the ML. The attached script produces different
> results for same initial simulation. By running it, it will produce 3 saved
> final simulations and 3 gnuplot plots (suffixed -00, -01, -02). They are
> different. It should be found out why.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/yade/+bug/394687/+subscribe
>

[Václav Šmilauer (eudoxos)]

In that case it is a bug in your script, not in yade; the try...except block will stop execution of commands when an exception is raised, going to the except clause directly.

[Anton Gladky (gladky-anton)]

I have added the tarball with several files.

If you start test.py just by normal way 'yade-trunk test.py' the last result string is 20914 (almost total output is in 'notable').
If the same script is started with yade-multi and multi-table, the last result string is 21068 (test.test1.log and test.test1.log).
If the 'DISPLAY=' word is deleted from yade-multi in line 231, results from multi and normal start are the same.

The script is tested with the latest r2145.

Thank you.

[Václav Šmilauer (eudoxos)]

This is separate bug from the first one. The cause is that opening the gui Controller right after simulation starts changes timestep (it truncates decimal points, so you have 4e-7 instead of 4.67e-7). I will try to fix that somehow.

[Václav Šmilauer (eudoxos)]

Fixed in r2079.

[Anton Gladky (gladky-anton)]

Thanks a lot! Now it is working reliable!

[Anton Gladky (gladky-anton)]

Sorry, still have a problem with -multi.

Results were the same because the bug with interactions was fixed and we did not see the difference in number of cohesive contacts. But still there is a difference in other parameters.

I will try to add test-script for that.

[Václav Šmilauer (eudoxos)]

Please file separate bug for that.

[Anton Gladky (gladky-anton)]

I was wrong, -multi works the same as standalone. Sorry.