Sometimes installation master node hangs

[Nastya Urlapova]

iso #205

Start installation master node

11853 0.0 0.0 11300 236 ? S 06:46 0:00 | \_ /bin/bash -e /usr/local/bin/start.sh
root 15031 0.0 0.0 22180 96 ? Ss 06:50 0:00 | \_ xinetd -stayalive -pidfile /var/run/xinetd.pid
root 15682 0.0 0.0 22180 784 ? S 06:52 0:00 | \_ /usr/sbin/xinetd -dontfork
root 11973 0.0 0.0 18904 740 ? Ss 06:46 0:00 \_ lxc-start -n d44a3c7c43b879919a247afdc3e21621000714b1c0864c96c5435a
root 11982 0.0 0.0 11300 64 ? S 06:46 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 12039 0.7 1.7 228996 17556 ? Dl 06:46 0:17 | \_ /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/
root 12184 0.0 0.0 18904 740 ? Ss 06:46 0:00 \_ lxc-start -n 37898a810b35ee1f6ecdbbe14551c03778414074d1690837508fdd
root 12192 0.0 0.0 11300 72 ? S 06:46 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 12276 0.8 1.0 303952 11220 ? Dl 06:46 0:20 | \_ /usr/bin/ruby /usr/bin/puppet apply -d -v /etc/puppet/modul
root 12961 0.0 0.0 18904 740 ? Ss 06:46 0:00 \_ lxc-start -n 8e8c403c9470aefe29b213c64ca00f25bf554ca705a284ffe21369
root 12974 0.0 0.0 11300 84 ? S 06:46 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 13039 0.8 0.8 236564 8400 ? Dl 06:46 0:22 | \_ /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/
root 13119 0.0 0.0 18904 740 ? Ss 06:47 0:00 \_ lxc-start -n 22029007db0c4d379c8cfc8ff0b6e2a99e61a87772bdc4dcfaedc3
root 13128 0.0 0.0 11300 84 ? S 06:47 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 13193 0.9 3.5 301336 35904 ? Dl 06:47 0:22 | \_ /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/
root 13297 0.0 0.0 18904 828 ? Ss 06:47 0:00 \_ lxc-start -n e64269cdbcff155a92dfde0cbbf61bcdedc1b2fb71466a72fb37dd
root 13302 0.0 0.0 11300 80 ? S 06:47 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 13400 0.8 1.4 233700 14864 ? Dl 06:47 0:20 | \_ /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/
root 13854 0.0 0.0 18904 828 ? Ss 06:47 0:00 \_ lxc-start -n 72a58c61dc01f3e24031cd847c26228d577de0e2941684208aba90
root 13871 0.0 0.0 11300 72 ? S 06:47 0:00 | \_ /bin/bash /usr/local/bin/start.sh
root 14003 0.7 1.2 233700 12280 ? Dl 06:47 0:18 | \_ /usr/bin/ruby /usr/bin/puppet apply -v /etc/puppet/modules/
root 14300 0.0 0.0 18904 740 ? Ss 06:48 0:00 \_ lxc-start -n 54a54cd82e40fccf0f5decfb155c30e4619761232df94a0590f643
root 14310 0.0 0.0 11300 84 ? S 06:48 0:00 \_ /bin/bash /usr/local/bin/start.sh
root 14430 0.7 1.7 228608 17700 ? Dl 06:48 0:17

see screenshots

[Sergii Golovatiuk (sgolovatiuk)]

This node doesn't have recommended kernel

[root@nailgun ~]# uname -r
2.6.32-431.el6.x86_64

I see a lot of errors in docker log file due to this kernel version. ISO should be rebuilt with recommended kernel.

[Aleksandr Didenko (adidenko)]

All pupper processes listed above are in "D" state:

D - uninterruptible sleep (usually IO)

Are you sure the server this deployment was running on had no performance issues at that time? Especially with IO subsystem.

[Igor Shishkin (teran)]

We just checked out IO load in real time, %util value was about 50% so I don't thing it's about performance.

[Vladimir Kuklin (vkuklin)]

is this reproduced in system tests environment or only during local deployment?

[Matthew Mosesohn (raytrac3r)]

The ps command is clipping off the end. Puppet is still running on 6 containers. You can try to run `docker ps | xargs -n1 docker logs` and capture the output and see if there are any interesting messages.

[Aleksandra Fedorova (bookwar)]

I've attached screen with logs from this build:

http://jenkins-product.srt.mirantis.net:8080/view/fuellib_test/job/fuellib_test.admin_node/210/

Blue terminal shows dstat on the host system, white shows 'docker ps' on master node and grey shows top from master node as well.

There is no real load on the host system, while swap is used inside the master node.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93888

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93888
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=c7200f53a3c9b2826206e4b361c5e35c0156f6da
Submitter: Jenkins
Branch: master

commit c7200f53a3c9b2826206e4b361c5e35c0156f6da
Author: NastyaUrlapova <email address hidden>
Date: Fri May 16 14:40:12 2014 +0400

    Increase memory size on master node

    Closes-Bug: #1319678
    Change-Id: I09d1567251175e6a5b42163fd4d55aa8facd9a01

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/93902

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93902
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=618597b18aaa9b4ad451dcd455e699971cf7582f
Submitter: Jenkins
Branch: master

commit 618597b18aaa9b4ad451dcd455e699971cf7582f
Author: Mike Scherbakov <email address hidden>
Date: Fri May 16 16:33:52 2014 +0400

    Increased RAM size for master node VM to 1.5Gb

    With introduction of Docker, master node requirements
    are increased and 1Gb of RAM doesn't seem to be enough:
    Linux starts to use swap. To avoid this and speed up
    master node boostrapping, increasing default RAM size.

    Change-Id: Ib6ed191b49729f480388b0414c8407571242ce84
    Related-Bug: 1319678

[Nastya Urlapova (aurlapova)]

installation of master node already stable.