Swift-container-sync failed to start that leads to deployment in error status

[Tatyanka]

steps:
1. create cluster:
  - 1 controller + 2 compute + 1 cinder
2. run network verification
3. Deploy it
4. Run ostf
5. Add 2 controllers, run network checker
6. Deploy changes

Actual result:
Deployment of new controllers failed with error in puppet:
http://paste.openstack.org/show/464331/

in swift-all.log from one of the failed node :
http://paste.openstack.org/show/464332/
Expected:
Redeployment pass without error, cluster is ready

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "288"
  build_id: "288"
  nailgun_sha: "93477f9b42c5a5e0506248659f40bebc9ac23943"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "121016a09b0e889994118aa3ea42fa67eabb8f25"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

[Tatyanka (tatyana-leontovich)]

Snapshot available https://drive.google.com/a/mirantis.com/file/d/0B_tSitrwrgvoVnFFV054OF9JODA/view?usp=sharing, failed nodes: node-7 and node-8

[Matthew Mosesohn (raytrac3r)]

There is a connectivity issue from node-8 to node-6 on storage VLAN. primary-controller set up swift okay, but secondary controllers (node-7, node-8) can't reach 10.109.4.6 (node-6 storage IP). Since Tatyana's env is erased, we need to get another env to reproduce this. Marking incomplete

[Tatyanka (tatyana-leontovich)]

Two deployment of this case is passed / one failed with keystone issue, so for now move to invalid.

[Ivan Berezovskiy (iberezovskiy)]

This issue was reproduced several times in this job: https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_tun_scale

Last failed build is 68.

Errors from puppet.log:

http://paste.openstack.org/show/481612/

http://paste.openstack.org/show/481611/

From swift-all.log:

http://paste.openstack.org/show/481613/

[Matthew Mosesohn (raytrac3r)]

We still need a revertable env. I missed this because I was on PTO. Back to incomplete until we have an env

[Matthew Mosesohn (raytrac3r)]

Still incomplete until we have an environment available.

[Maciej Relewicz (rlu)]

Due to a month inactivity switched to invalid.