docker-puppet.py: rabbitmq -q list_users run during stack update

Bug #1737174 reported by James Slagle on 2017-12-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
James Slagle

Bug Description

If there's been a configuration change to rabbitmq that causes docker-puppet.py to reconfigure rabbitmq during a stack-update, it could fail because the puppet manifest uses "rabbitmqctl -q list_users". It will fail if rabbitmq is not running. This is triggered in the scenario where there is an initial failed deployment prior to rabbitmq starting, the user does a necessary change and attempts to redeploy, which is a stack-update now.

The update triggers this from puppet-tripleo/manifests/profile/base/rabbitmq.pp:

  if $step >= 2 {
    # In case of HA, starting of rabbitmq-server is managed by pacemaker, because of which, a dependency
    # to Service['rabbitmq-server'] will not work. Sticking with UPDATE action.
    if $stack_action == 'UPDATE' {
      # Required for changing password on update scenario. Password will be changed only when
      # called explicity, if the rabbitmq service is already running.
      rabbitmq_user { $rabbitmq_user :
        password => $rabbitmq_pass,
        provider => 'rabbitmqctl',
        admin => true,
      }
    }

Even though we have noop'd Rabbitmq_user, for some reason, the above resource still runs during a puppet "prefetch" step.

The debug output from docker-puppet.py ends up looking like:

Debug: Prefetching rabbitmqctl resources for rabbitmq_user
Debug: Executing: '/usr/sbin/rabbitmqctl -q list_users'
Debug: Command failed, retrying

which eventually times out and fails the docker-puppet.py run after 3 minutes.

Changed in tripleo:
status: New → In Progress
importance: Undecided → High
assignee: nobody → James Slagle (james-slagle)
milestone: none → queens-3
James Slagle (james-slagle) wrote :

I suspect that during any stack-update where docker-puppet.py reruns for rabbitmq we are running "rabbitmqctl -q list_users", the problem is just usually not seen since rabbitmq had been previously started.

James Slagle (james-slagle) wrote :

We noop'd the rabbitmq_user resource in https://review.openstack.org/#/c/524751/, but apparently that is not working for some reason.

Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers