[Update] Signon to CIB failed: Transport endpoint is not connected

[Ruslan Khozinov]

Detailed bug description:
Upgrade cluster from 9.1 to 9.2 (deb http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-17-010321 mos9.0-proposed main restricted) failed on Task[pkg_upgrade/7], Task[openstack-controller-contrail/8] tasks.

Steps to reproduce:
   - Deploy 9.1 env with following nodes:
     * compute_cinder
     * contrail-analytics-db
     * contrail-controller
     * controller
     * contrail-controller_contrail-analytics
     * contrail-analytics-db_contrail-analytics
     * contrail-analytics
     * contrail-controller
     * contrail-analytics-db
   - Add proposed repo for env (mos-proposed: "deb http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-17-010321 mos9.0-proposed main restricted" with 1200 priority)
   - Add to Maintenance update: Repos for upgrade mos-proposed
   - Download mos-mu tool on master (git clone https://github.com/aepifanov/mos_mu.git)
   - Make a perform a preparation playbook (ansible-playbook playbooks/mos9_prepare.yml -e '{"env_id":<env_id>}')
   - Update fuel node (ansible-playbook playbooks/update_fuel.yml)
   - Update env (fuel2 update --env <ENV_ID> install)

Observed behavior:
Deployment has failed:
(/Stage[main]/Contrail::Controller/Service[heat-engine]) Could not evaluate: Execution of '/usr/sbin/cibadmin -Q' returned 107: Signon to CIB failed: Transport endpoint is not connected

pcs status returned:
Error: cluster is not currently running on this node

pacemaker log:
Nov 21 10:25:21 [15025] node-8.test.domain.local pacemakerd: warning: mcp_read_config: Could not connect to Cluster Configuration Database API, error 2
Nov 21 10:25:21 [15025] node-8.test.domain.local pacemakerd: notice: main: Could not obtain corosync config data, exiting
Nov 21 10:25:21 [15025] node-8.test.domain.local pacemakerd: info: crm_xml_cleanup: Cleaning up memory from libxml2
Nov 21 10:25:21 [15998] node-8.test.domain.local pacemakerd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
Nov 21 10:25:21 [15998] node-8.test.domain.local pacemakerd: info: mcp_read_config: cmap connection setup failed: CS_ERR_LIBRARY. Retrying in 1s
Nov 21 10:25:22 [15998] node-8.test.domain.local pacemakerd: info: mcp_read_config: cmap connection setup failed: CS_ERR_LIBRARY. Retrying in 2s
Nov 21 10:25:24 [15998] node-8.test.domain.local pacemakerd: info: mcp_read_config: cmap connection setup failed: CS_ERR_LIBRARY. Retrying in 3s
Nov 21 10:25:27 [15998] node-8.test.domain.local pacemakerd: info: mcp_read_config: cmap connection setup failed: CS_ERR_LIBRARY. Retrying in 4s
Nov 21 10:25:31 [15998] node-8.test.domain.local pacemakerd: info: mcp_read_config: cmap connection setup failed: CS_ERR_LIBRARY. Retrying in 5s

cinder can't connect to the db:
2016-11-21 10:28:50.844 8141 WARNING oslo_db.sqlalchemy.engines [req-9918c9c6-6246-4613-8a00-71c46e37d7a3 - - - - -] SQL connection failed. -39 attempts left

nova can't connect to the db:
2016-11-21 10:30:05.927 6523 WARNING oslo_db.sqlalchemy.engines [req-f14cb37a-e0d0-4c21-a378-d8d8e0800172 - - - - -] SQL connection failed. -322 attempts left.
Diagnostic snapshot: https://drive.google.com/open?id=0B8nyPqe6rrN1MU15Nlc4dDZhZFk

[Ilya Bumarskov (ibumarskov)]

Seems, it's duplicate of https://bugs.launchpad.net/fuel/+bug/1641947