Unable to update 9.0 cluster with local mirrors - "Unable to install rabbitmq-server_3.6.5-1~u14.04+mos1_all.deb "

[Artem Hrechanychenko]

Detailed bug description:
2016-09-15 08:29:58 +0000 Package[rabbitmq-server](provider=apt_fuel) (warning): Attempt '1' of '3' has failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install rabbitmq-server' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
 cloud-guest-utils eatmydata python-cheetah python-oauth python-serial
 python3-pycurl python3-software-properties software-properties-common
 unattended-upgrades
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
 rabbitmq-server
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 5324 kB of archives.
After this operation, 398 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
 rabbitmq-server
Authentication warning overridden.
Get:1 http://10.109.32.2:8080/mirrors/mos-repos/ubuntu/snapshots/9.0-2016-09-14-182322/ mos9.0-proposed/main rabbitmq-server all 3.6.5-1~u14.04+mos1 [5324 kB]
Fetched 5324 kB in 0s (19.1 MB/s)
(Reading database ... 111077 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.6.5-1~u14.04+mos1_all.deb ...
Unpacking rabbitmq-server (3.6.5-1~u14.04+mos1) over (3.6.1-1~u14.04+mos3) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up rabbitmq-server (3.6.5-1~u14.04+mos1) ...

HiPE compiling: |---------------------------------------------------------|
                |#############Terminated
dpkg: error processing package rabbitmq-server (--configure):
subprocess installed post-installation script returned error exit status 143
Errors were encountered while processing:
rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1

Steps to reproduce:
1) Deploy 9.0 cluster using locall mirrors:
   1.1 fuel-mirror create -P ubuntu -G mos
   1.2 fuel-mirror create -P ubuntu -G ubuntu
   1.3 change mirrors in UI to local mirrors
   1.4 deploy cluster
2) add latest snapshot of mos/centos repo
3) install python-cudet package
4) add local mirror of latest ubuntu snapshot for mos9.0-proposed
5) add local proposed repo to cluster settings
6) deploy changes
7) run "update-prepare prepare master, update-prepare prepare env env_id, update-prepare update master"
8) run "fuel2 updates --env env_id --repos test-proposed --restart-rabbit --restart-mysql install

Expected results:
 cluster updated successfully

Actual result:
Deployment has failed. All nodes are finished. Failed tasks: Task[pkg_upgrade/1] Stopping the deployment process!

 dpkg: error processing package rabbitmq-server (--configure):
subprocess installed post-installation script returned error exit status 143

[Timur Nurlygayanov (tnurlygayanov)]

Could you please attach the diagnostic snapshot?

Thank you!

[Roman Vyalov (r0mikiam)]

the package is placed in the repository , reassign to mos-linux team
http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-09-14-182322/pool/main/r/rabbitmq-server/rabbitmq-server_3.6.5-1~u14.04%2bmos1_all.deb

[Ivan Suzdal (isuzdal)]

I can't reproduce it. Could you provide access to failed env?

[Artem Hrechanychenko (agrechanichenko)]

cannot reproduce, invalid repo name in --repo argument. checked, cluster was updated

[Zoltan Szeder (zoltan-szeder)]

I had the same problem, and it was caused by the fuel-rabbitmq-fence service.
It was unable to reach the rabbitmq service and started to fence it. Since the HiPE compile is run by a rabbitmq process, it became a victim for the termination.

To workaround the problem, we did the following:
1. Stop the fuel-rabbitmq-fence service
2. Update the rabbitmq-server package
3. Start the fencing service

It does not occur since then.

[Nikita Koshikov (nkoshikov)]

The original problem in this issue is sequence of upgrading.
Because of fuel-ha-utils package upgraded - that brings new ocf code for rabbitmq. The status command was changed and is not compatible with 3.6.1 rabbit. After that pacemaker starts to think that rabbit is dead - because monitor command returns error and tries to restart cluster.

To properly fix this - either you should upgrade rabbitmq-server first and than fuel-ha-utils, or set OCF_RESKEY_rmq_feature_health_check == 'false', that will trigger node_health_check_legacy() function to do verification - that supports both version.