[Update] Update cluster from 9.0 to 9.2 failed on upgrade package [pacemaker-cli-utils]

[Ilya Bumarskov]

Fuel 9.2 snapshot-id#724

Steps to reproduce:
   - Deploy 9.0 env (deployed via fuel-qa with FORCE_DISABLE_UPDATES=True) with following nodes:
         * Controller
         * Compute, cinder
         * Compute, cinder
   - Install mos-playbooks tool on master node (git clone https://github.com/aepifanov/mos_mu.git && git checkout 9.2 && cd mos_mu)
   - Execute preparation playbooks (ansible-playbook playbooks/mos9_prepare_fuel.yml && ansible-playbook playbooks/mos9_prepare_env.yml -e '{"env_id":1, "snapshot_repo":"<MOS_UBUNTU_MIRROR_ID>"}')
   - Update fuel node (ansible-playbook playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}')
   - Update env (fuel2 update install --env 1 --repos mos9.2)

Observed behavior:
Deployment has failed. All nodes are finished. Failed tasks: Task[pkg_upgrade/3] Stopping the deployment process!

Puppet logs (fragment):
(/Stage[main]/Osnailyfacter::Upgrade::Pkg_upgrade/Package[pacemaker-cli-utils]/ensure) change from 1.1.14-2~u14.04+mos1 to 1.1.14-2~u14.04+mos3 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install pacemaker-cli-utils' returned 100: Reading package lists...

[Anton Chevychalov (achevychalov)]

Systemd trouble again?

Puppet.log on node-3
====================
Failed to issue method call: Unit crm_mon.service not loaded.
dpkg: error processing archive /var/cache/apt/archives/pacemaker-cli-utils_1.1.14-2~u14.04+mos3_amd64.deb (--unpack):
 subprocess new pre-removal script returned error exit status 5
Failed to issue method call: Unit crm_mon.service failed to load: No such file or directory. See system logs and 'systemctl status crm_mon.service' for details.
Errors were encountered while processing:
 /var/cache/apt/archives/pacemaker-cli-utils_1.1.14-2~u14.04+mos3_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
2017-01-09 08:05:44 +0000 Puppet (notice): Finished catalog run in 22.03 seconds

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/29679

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29679
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 7d9d4b8dc5d4b56816aedf9825eddd75e7d1fde0
Author: Ivan Suzdal <email address hidden>
Date: Mon Jan 9 11:15:01 2017

Disable systemd related stuff in d/rules.

Removing *.service files in override_dh_install
doesn't look like best idea, but imho it seems to be
more clear than patching makefiles.

Change-Id: I7bbad688d3d1980bec133748f5821fee70a8b69f
Closes-Bug: #1654966

[Ilya Bumarskov (ibumarskov)]

Fix doesn't help. Bug is reproducible on snapshot-id #743

Node with controller role:
root@node-3:~# apt-cache policy pacemaker
pacemaker:
  Installed: 1.1.14-2~u14.04+mos4
  Candidate: 1.1.14-2~u14.04+mos4
  Version table:
 *** 1.1.14-2~u14.04+mos4 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2017-01-10-112420/ mos9.0-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.14-2~u14.04+mos1 0
       1050 http://10.109.5.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
     1.1.10+git20130802-1ubuntu2.3 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.1.10+git20130802-1ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418397
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f6a890bfdf1a9f7e23d62ab4b7635a933dbd9fd2
Submitter: Jenkins
Branch: stable/mitaka

commit f6a890bfdf1a9f7e23d62ab4b7635a933dbd9fd2
Author: Ivan Suzdal <email address hidden>
Date: Tue Jan 10 17:00:34 2017 +0300

    Add create-policy-rc.d for pacemaker-cli-utils package

    Change-Id: I8308e5f8f7158c893298e1ad2ae64b3ce3ff2b81
    Closes-Bug: #1654966

[Ilya Bumarskov (ibumarskov)]

Fix doesn't help. Reproduced on snapshot-id #746.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418850

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418850
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=aa775640218aa844f68081d2eaa71253b23c33c0
Submitter: Jenkins
Branch: stable/mitaka

commit aa775640218aa844f68081d2eaa71253b23c33c0
Author: Ivan Suzdal <email address hidden>
Date: Wed Jan 11 14:08:04 2017 +0300

    Change create-policy-rc.d for crm_mon service

    Change-Id: I40b21c2ee8ebab741e320924e3ec27b450235a7c
    Closes-Bug: #1654966

[Ilya Bumarskov (ibumarskov)]

Verified on snapshot-id $748

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421342

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421346

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421390

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421410

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/421390

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/421342

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/421346

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/421410