rpc CrashLoopBackOff after upgrade from mitaka to newton

[Sergey Galkin]

Steps to reproduce
1. Deploy ccp with configs from https://bugs.launchpad.net/fuel-ccp/+bug/1682429 on the 95 nodes
2. Try to run shaker (http://pyshaker.readthedocs.io/en/latest/) scenario openstack/full_l2 with versions.yaml-mitaka as versions.yaml
3. When heat stack was created change versions.yaml to versions.yaml-mitaka and execute 'ccp deployment'

All rabbit pods except one have a lot of restarts

root@node4:~# kubectl -n ccp get pods -o wide | grep -E '(rpc|notific)'
notifications-939738848-cp56s 1/1 Running 6 2h 10.249.164.172 node7
notifications-939738848-vd82m 1/1 Running 9 2h 10.244.225.92 node8
notifications-939738848-zm9xq 1/1 Running 0 2h 10.240.73.211 node6
rpc-1612787118-0j7th 1/1 Running 7 2h 10.244.225.109 node8
rpc-1612787118-0pzrp 1/1 Running 8 2h 10.249.164.144 node7
rpc-1612787118-k6vxz 1/1 Running 8 2h 10.240.73.216 node6

with one error

2017-04-13 15:32:09.410 - __main__ - DEBUG - Executing cmd:
/opt/ccp/bin/rabbitmq-readiness.sh
Traceback (most recent call last):
  File "/opt/ccp_start_script/bin/start_script.py", line 713, in <module>
    main()
  File "/opt/ccp_start_script/bin/start_script.py", line 576, in main
    do_status(args.role)
  File "/opt/ccp_start_script/bin/start_script.py", line 613, in do_status
    run_probe(readiness_probe)
  File "/opt/ccp_start_script/bin/start_script.py", line 583, in run_probe
    run_cmd(probe["command"])
  File "/opt/ccp_start_script/bin/start_script.py", line 372, in run_cmd
    raise ProcessException(proc.returncode)
__main__.ProcessException: Command exited with code 1

  31m 31m 1 {kubelet node6} spec.containers{rabbitmq} Warning Unhealthy Readiness probe failed: 2017-04-13 15:36:59.708 - __main__ - INFO - Getting global variables from /etc/ccp/globals/globals.json

Output of kubectl -n ccp describe pod rpc-1612787118-k6vxz and notifications-939738848-vd82m attached

[Sergey Galkin (sgalkin)]

This is network hi load issue