Comment 1 for bug 675874

Once you've run out of memory and started hitting swap, the memory previously used as cache for other running programs will be dropped, or inactive pages paged out to disk.

This does make things very slow and unresponsive for a while, but the system _should_ recover. Did your system recover eventually, or was it unresponsive until you rebooted?

Can you reproduce the issue every time yuo try the calculation?