so i made small investigation about this:

1) Locally deployed bvt-like environment (3 controllers, 3 Computes + Ceph, 2cpu, 3G, radosgw)
ISO - http://seed.fuel-infra.org/fuelweb-iso/fuel-10.0-community-704-2016-09-22_23-14-00.iso
   Ran OSTF tests on it (only one test failed - "Update stack actions: inplace, replace and update whole template"), but the bug is reproduced successfully, after OSTF mysqld was explicitly killed on 2 controllers, on third one it was not running (no message in logs that it was killed ), moreover on one of the controllers even nova-api was killed:

<3>Sep 26 13:00:04 node-5 kernel: [ 9758.142888] Killed process 5669 (mysqld) total-vm:2533636kB, anon-rss:102244kB, file-rss:0kB
<3>Sep 26 13:30:05 node-5 kernel: [11558.395347] Killed process 11566 (nova-api) total-vm:762948kB, anon-rss:138896kB, file-rss:1984kB
<3>Sep 26 13:00:12 node-4 kernel: [ 9766.350272] Killed process 18205 (nova-api) total-vm:761180kB, anon-rss:141460kB, file-rss:3276kB
<3>Sep 26 13:03:08 node-4 kernel: [ 9942.369298] Killed process 21134 (mysqld) total-vm:2570044kB, anon-rss:106276kB, file-rss:0kB
<3>Sep 26 13:30:04 node-4 kernel: [11558.377802] Killed process 18209 (nova-api) total-vm:759576kB, anon-rss:140708kB, file-rss:1604kB

   Resource usage before OSTF(it is similar on all controllers):
              total        used        free      shared  buff/cache   available
Mem:           2.9G        2.6G        114M         69M        252M        105M
Swap:          3.0G        2.3G        766M

   Resource usage during OSTF(it is similar on all controllers):
              total        used        free      shared  buff/cache   available
Mem:           2.9G        2.7G         79M         64M        153M         22M
Swap:          3.0G        3.0G          0B

2) Locally deployed smoke_neutron-like environment (1 controller, 2 Computes + Cinder, 2cpu, 3G)
ISO - http://seed.fuel-infra.org/fuelweb-iso/fuel-10.0-community-704-2016-09-22_23-14-00.iso
   Ran OSTF tests on it, and reproduced this bug, mysqld was killed, all symptoms are the same as in bvt-like case except that more OSTF tests failed
   
Summary:

In both cases - I checked also if some process could eat anomalous memory volume, and could not find such a process, it looks like all processes in summary are eating it [1]. If you look at mysql memory usage it is not high, because real allocated memory is RSS and it is always around 100M or during OSTF it is 150 MB. Its killed only because of OOM killer algorythm.

So looks like indeed all issues are related to lack of RAM on fuel slave nodes. Currently this issues are being reproduced (on different CIs[2,3] and mostly on non-HA cases). It is not permanent but floating issue, I guess high IO operations speed and overall low load level of CI slave allow sometimes eliminate this issue. I had not the chance yet to look at 9.1 case but in 10.0 we are using Ubuntu 16.04 and in 9.1 we are still on Ubuntu 14.04. It can make great difference because of OS memory usage. Will try soon to take a look at 9.1 

[1] - http://paste.openstack.org/show/582970/
[2] - https://product-ci.infra.mirantis.net/job/10.0.main.ubuntu.smoke_neutron/712/
[3] - https://ci.fuel-infra.org/view/BVT/job/10.0.repos.main.ubuntu.smoke_neutron/99