[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-02-07_20-50-01
build_number: '76'
feature_groups:
mirantis
fuellib_sha: 64f3ebe9fcbd18bf6c80a948e06061783a090347
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 2ef819732a3ee7acf7b610e7d1c1a6da0434c1a0
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
release_versions:
2014.2-6.0.1:
VERSION:
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
build_id: 2015-02-07_20-50-01
build_number: '76'
feature_groups:
mirantis
fuellib_sha: 64f3ebe9fcbd18bf6c80a948e06061783a090347
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 2ef819732a3ee7acf7b610e7d1c1a6da0434c1a0
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
Baremetal,Ubuntu, HA, Neutron-gre,Ceilometer,Ceph-all, Debug, 6.0.1_76
Controllers:3 Computes:97
deployment was successful
rally light was succssfulhttp://mos-scale.vm.mirantis.net/test_results/build_6.0.1-76/jenkins-10_env_run_rally_light-15-ubuntu-MSK-2015-02-13-20:26:27-2015-02-13-22:34:22/rally_report.html
But after rally ull test I have found the following bihaviour on the primary controller:
root@node-32:~# ls -la
-bash: fork: Cannot allocate memory
root@node-32:~# top
-bash: fork: Cannot allocate memory
root@node-32:~# less /var/log/messages
-bash: fork: Cannot allocate memory
root@node-32:~# dmesg
-bash: fork: Cannot allocate memory
I could execute the following command, but after that i lost ssh connection:
root@node-32:~# exec ls
ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph.conf ceph.log ceph.mon.keyring openrc
Connection to node-32 closed.
I couldn't open nw ssh connection. I couldn't reach bash of the node even ipmi. (screenshot is attached)
I've rebooted the node at Feb 15 17:39:33 UTC. from messages (I can see log files after rebooting):
<6>Feb 15 17:39:33 node-32 kernel: imklog 5.8.6, log source = /proc/kmsg started.
latest message before rebooting:
<134>Feb 14 13:55:54 node-32 haproxy[11311]: 192.168.0.34:52990 [14/Feb/2015:13:14:34.260] mysqld mysqld/node-32 1/0/2480311 1015 – 1141/1130/1130/1130/0 0/0
Now can reach ivia ssh and execute bash cmmands
from messages log:
Feb 15 18:39:49 node-32 kernel: [ 3650.131654] perf samples too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
The latest strings from keystone:
<15>Feb 14 13:55:54 node-32 keystone-all Auth token not in the request header. Will not build auth context.
<15>Feb 14 13:55:54 node-32 keystone-all arg_dict: {}
I have no idea how I can know why node was hanged
Diagnostic snapshot is here: https://drive.google.com/a/mirantis.com/file/d/0Bx4ptZV1Jt7helc3OFh3T19WcFk/view?usp=sharing
We will require an environment. Can you please collaborate on skype?