error at the end of deployment: restart_rados.sh not found on puppet

[Lope Beltran II]

Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && sh /etc/puppet/modules/osnailyfacter/modular/astute/restart_rados.sh

Basically at the end of the deployment, puppet cant seem to find the restart_rados.sh on all the controllers resulting to the error of the deployment.

All my repos seems to be up to date.

Cant upload the snapshot, i left the deployment overnight after a few deployments and the snapshot reached 500+mb.

[Lope Beltran II (beltranlope)]

Deplyoment: 6.1 CentOS HA (nova-network vlan)

4 controller
15 compute
2 ceph
3 mongodb

installed with zabbix.

[Lope Beltran II (beltranlope)]

This is the snapshot. Tested without zabbix.

[Alex Schultz (alex-schultz)]

So restart_rados.sh has been replaced with a puppet equivalent as part of https://review.openstack.org/#/c/193246/. Which I believe was part of update 1. How did you create your environment? You may need to resync the tasks after updating to have it correctly replace the restart rados task with the new one. This probably should be done as part of the fuel-library rpm update if one was applied. Was the fuel master updated with any updates?

[Lope Beltran II (beltranlope)]

I may have done a yum update when i installed the zabbix plugin.

Can you post the commands i need to do after the yum update so as to resync my tasks?

[Alex Schultz (alex-schultz)]

After updating the fuel-library package (and if you do a blanket yum update this may happen), run on the fuel master:

  fuel release --sync-deployment-tasks --dir /etc/puppet/

I'll also take a look to see if we do this as part of a post-update in the package but my assumption based on this is that we are not doing that so you'd need to do this. As part of the newer fuel-plugin-builder we are allowing a post install script for plugins so this would be a prime place to do this. Especially if you are adding tasks in your plugin itself.

[Lope Beltran II (beltranlope)]

Thanks it worked! But the environment have errors. It seems the active controller has keystone service working but its database seems to be encountering cluster issue. I'm getting keystone-client errors. Even in the health check, all failed because of the keystone seems to be unreachable. Even zabbix encountering problems communicating with the mysql server.

My setup was:
1. installed fuel 6.1
2. yum update -y
3. fuel release --sync-deployment-tasks --dir /etc/puppet/
4. installed zabbix plugin
5. Created a new environment (CentOS HA Deplyoment using Nova-network - 4 controllers, 16 compute, 2 ceph storage, 3 mongodb)
6. Successful Deplyoment but has issues on keystone client. Can't login to horizon, zabbix cant reach mysql database and health checks failed due to keystone-client error.

[Lope Beltran II (beltranlope)]

I did crm(live)resource# status and it seems the vip__management and zabbix-server keeps failing. I tried cleaning them up but after a few minutes it will fail again. I already did a reset on my environment so the logs are gone sorry. I'm on a tight deadline on my experiment, but i think the setup above can reproduce the same problem.

[Alex Schultz (alex-schultz)]

Lope, can you open a separate bug for the keystone-client errors? I'd like to keep this one just for the radosgw issue.

[Alex Schultz (alex-schultz)]

Documentation update done as part of https://review.fuel-infra.org/#/c/9627

[Michael Petersen (mpetason)]

The issue may be coming up due to 4x controllers. There should be 3x controllers for the installation, unless the 4th controller is Zabbix. On the Installations it looks like there are 4x controllers configured, which could cause issues with clustering.

[Roman Rufanov (rrufanov)]

customer found on 6.1

[Alex Schultz (alex-schultz)]

This bug about the lack of restart_rados.sh only affects the 6.0 to 6.1 upgrade because the file was removed as part of 6.1. This is an issue with the upgrade process not syncing the tasks on upgrades.

[Matthew Mosesohn (raytrac3r)]

This affects fuel-upgade. Tasks should be synced when updating from one release to another. If this is not valid, then please update status accordingly.

[Ihor Kalnytskyi (ikalnytskyi)]

Hey guys,

I'm sorry, but I can't figure out why it's assigned to fuel-python and how we (as fuel-python) can help?

Could you please describe what should be done from our side to make users happy? :)

[Matthew Mosesohn (raytrac3r)]

I think there was an issue with tasks syncing or with packaging, but it's not entirely clear here. We need to reproduce this and get a diagnostic snapshot

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/231584

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/231584
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9821477325a6069a88be2f7086981de5b3f909fa
Submitter: Jenkins
Branch: master

commit 9821477325a6069a88be2f7086981de5b3f909fa
Author: Matthew Mosesohn <email address hidden>
Date: Tue Oct 6 19:38:46 2015 +0300

    Sync deployment tasks on fuel-library upgrade

    Simple yum update could break redeployments because
    a given task could be changed or deleted. Therefore,
    an attempt or warning should be made to update
    deployment tasks.

    Added explicit dependency on fuelclient.

    Change-Id: I06a905aa0bbaefbeb7cbdc60b563f11575bd5bd8
    Closes-Bug: #1475530

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/234756

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/234757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234757
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e29a3d9ef8b9001f274c1668bd00c5f25960a3df
Submitter: Jenkins
Branch: master

commit e29a3d9ef8b9001f274c1668bd00c5f25960a3df
Author: Matthew Mosesohn <email address hidden>
Date: Wed Oct 14 15:22:48 2015 +0300

    Fix typo in sync deployment tasks

    Change-Id: Ic7d7fc50afcb3338212156cafa182c19edc8d71b
    Related-Bug: #1475530

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234756
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=044287f814e2f1ad35542d3e5a549e3f62262e59
Submitter: Jenkins
Branch: stable/7.0

commit 044287f814e2f1ad35542d3e5a549e3f62262e59
Author: Matthew Mosesohn <email address hidden>
Date: Wed Oct 14 15:21:27 2015 +0300

    Sync deployment tasks on fuel-library upgrade

    Simple yum update could break redeployments because
    a given task could be changed or deleted. Therefore,
    an attempt or warning should be made to update
    deployment tasks.

    Added explicit dependency on fuelclient.

    Change-Id: I76efd12e1262bc281856f0e927cdf2668f2ce532
    Cherry-Picked-From: I06a905aa0bbaefbeb7cbdc60b563f11575bd5bd8
    Closes-Bug: #1475530

[Andrey Sledzinskiy (asledzinskiy)]

verified on 7.0-301 with MU
checked scenario with fuel-library package update and checked that tasks were synced

[Andrii Petrenko (aplsms)]

This bug affected MOS 6.1 MU3
How to repeat:
1. install vanilla MOS 6.1
2. apply MOS 6.1 MU3.
3. add new environment or update existent
4. get the error:
Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && sh /etc/puppet/modules/osnailyfacter/modular/astute/restart_rados.sh

Workaround:
run by root after deployment update on the Fuel master node:

fuel release --sync-deployment-tasks --dir /etc/puppet/

After reapplying changes environment become operational and able to create instance.

[Denis Meltsaykin (dmeltsaykin)]

Note about this will be added to the Release Notes.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/6.1
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/16065

[Tatyanka (tatyana-leontovich)]

verified
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "466"
  build_id: "466"
  fuel-nailgun_sha: "f81311bbd6fee2665e3f96dcac55f72889b2f38c"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "6823f1d4005a634b8436109ab741a2194e2d32e0"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "fe03d887361eb80232e9914eae5b8d54304df781"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "727f7076f04cb0caccc9f305b149a2b5b5c2af3a"

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16065
Submitter: Olga Gusarenko <email address hidden>
Branch: stable/6.1

Commit: 96299a87133627e04589740f11239251833068a6
Author: Denis V. Meltsaykin <email address hidden>
Date: Wed Jan 13 10:06:01 2016

[6.1] MU4 additional changes

1. One new step was added to the Fuel Master update instruction, as it is
essential step to successful deployment after update under certain
circumstances.
2. Changed all occurences of `$` to `#`, as we run this commands as
root-user, not an ordinary one.

This CR must sctrictly be merged _after_
https://review.fuel-infra.org/#/c/14749/

Change-Id: I81148735d59799c60fd76ddf5a2fea077ce192ff
Closes-Bug: #1475530

[Vadim Rovachev (vrovachev)]

If we use next scheme:
1. Deploy vanilla 6.1 master node
2. Create and deploy cluster
3. Update master node using command:
 yum update -y && dockerctl destroy all && bootstrap_admin_node.sh
4. Dodeploy env
we have env with failled state.

If we use next scheme:
1. Deploy vanilla 6.1 master node
2. Create and deploy cluster
3. Update master node using command:
 yum update -y && docker load -i /var/www/nailgun/docker/images/fuel-images.tar && dockerctl destroy all && dockerctl start all && fuel release --sync-deployment-tasks --dir /etc/puppet/
4. Dodeploy env
we have env with operational state.