Stein upgrade failing on rabbitmq_init_bundle

Bug #1832588 reported by Jiří Stránský
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Jiří Stránský

Bug Description

logs from rabbitmq_init_bundle

Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method `[]' for nil:NilClass
Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\n (file & line not available)
Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name'; \n (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\n (file & line not available)
Notice: Compiled catalog for controller-0.localdomain in environment production in 0.89 seconds
Info: Applying configuration version '1560264976'
Notice: /Stage[main]/Rabbitmq::Config/File[/etc/rabbitmq]/owner: owner changed 'rabbitmq' to 'root'
Notice: /Stage[main]/Rabbitmq::Config/File[/etc/rabbitmq]/group: group changed 'rabbitmq' to 'root'
Notice: /Stage[main]/Rabbitmq::Config/File[/etc/rabbitmq/ssl]/ensure: created
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq-env.config]/ensure: defined content as '{md5}90b516d5432cfd43b3d7795674a554b8'
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq-inetrc]/ensure: defined content as '{md5}4b28755bf79e459a7dc0697afc3ec198'
Notice: /Stage[main]/Rabbitmq::Config/File[enabled_plugins]/ensure: defined content as '{md5}3d4c2ba1989ab83544dcbd9d8b791a8b'
Notice: /Stage[main]/Rabbitmq::Config/File[/etc/security/limits.d/rabbitmq-server.conf]/ensure: defined content as '{md5}1030abc4db405b5f2969643e99bc7435'
Info: Computing checksum on file /etc/rabbitmq/rabbitmq.config
Info: /Stage[main]/Rabbitmq::Config/File[rabbitmq.config]: Filebucketed /etc/rabbitmq/rabbitmq.config to puppet with sum e02e0bf34644ed2ad4a38fb64974c699
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq.config]/content: content changed '{md5}e02e0bf34644ed2ad4a38fb64974c699' to '{md5}cb60abcfdeef5e4ce729698ec952fb0d'
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq.config]/owner: owner changed 'rabbitmq' to 'root'
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq.config]/mode: mode changed '0644' to '0640'
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/File_line[rabbitmq-pamd-succeed]/ensure: created
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Property[rabbitmq-role-controller-0]/Pcmk_property[property-controller-0-rabbitmq-role]/ensure: created
Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20190611-8-1xkxt80 create failed: Error: unable to set attribute rabbitmq-role
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Property[rabbitmq-role-controller-1]/Pcmk_property[property-controller-1-rabbitmq-role]/ensure: change from 'absent' to 'present' faile
d: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20190611-8-1xkxt80 create failed: Error: unable to set attribute rabbitmq-role
Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20190611-8-16ircqg create failed: Error: unable to set attribute rabbitmq-role
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Resource::Bundle[rabbitmq-bundle]/Pcmk_bundle[rabbitmq-bundle]: Dependency Pcmk_property[property-controller-1-rabbitmq-role] has fail
ures: true
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Resource::Bundle[rabbitmq-bundle]/Pcmk_bundle[rabbitmq-bundle]: Dependency Pcmk_property[property-controller-2-rabbitmq-role] has fail
ures: true
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Property[rabbitmq-role-controller-2]/Pcmk_property[property-controller-2-rabbitmq-role]/ensure: change from 'absent' to 'present' faile
d: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20190611-8-16ircqg create failed: Error: unable to set attribute rabbitmq-role
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Resource::Bundle[rabbitmq-bundle]/Pcmk_bundle[rabbitmq-bundle]: Skipping because of failed dependencies
Notice: /Stage[main]/Rabbitmq::Config/Systemd::Service_limits[rabbitmq-server.service]/Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf]/File[/etc/systemd/system/rabbitmq-server.service.d]/ensure: cre
ated
Notice: /Stage[main]/Rabbitmq::Config/Systemd::Service_limits[rabbitmq-server.service]/Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf]/File[/etc/systemd/system/rabbitmq-server.service.d/90-limits.co
nf]/ensure: defined content as '{md5}0f5523d08a1441f44070144708e774c4'
Info: /Stage[main]/Rabbitmq::Config/Systemd::Service_limits[rabbitmq-server.service]/Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf]/File[/etc/systemd/system/rabbitmq-server.service.d/90-limits.conf
]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Info: Class[Systemd::Systemctl::Daemon_reload]: Unscheduling all events on Class[Systemd::Systemctl::Daemon_reload]
Info: Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf]: Unscheduling all events on Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf]
Info: Class[Rabbitmq::Config]: Unscheduling all events on Class[Rabbitmq::Config]
Info: Computing checksum on file /var/lib/rabbitmq/.erlang.cookie
Info: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/File[/var/lib/rabbitmq/.erlang.cookie]: Filebucketed /var/lib/rabbitmq/.erlang.cookie to puppet with sum 261fd2c5ff7b7afc8d9afccd48ea38ad
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/File[/var/lib/rabbitmq/.erlang.cookie]/content: content changed '{md5}261fd2c5ff7b7afc8d9afccd48ea38ad' to '{md5}2a613e2d7e7c22e819ae1c5619bde655
'
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Pacemaker::Resource::Ocf[rabbitmq]/Pcmk_resource[rabbitmq]: Skipping because of failed dependencies
Info: Class[Tripleo::Profile::Pacemaker::Rabbitmq_bundle]: Unscheduling all events on Class[Tripleo::Profile::Pacemaker::Rabbitmq_bundle]
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/Exec[rabbitmq-ready]: Skipping because of failed dependencies
Info: Creating state file /var/lib/puppet/state/state.yaml
Changes:
            Total: 15
Events:
          Success: 15
          Failure: 2
            Total: 17
Resources:
          Changed: 12
      Out of sync: 14
           Failed: 2
          Skipped: 21
            Total: 38
Time:
        File line: 0.00
             File: 0.06
   Config retrieval: 1.02
         Last run: 1560265632
    Pcmk property: 416.69
            Total: 655.09
Version:
           Config: 1560264976
           Puppet: 5.5.10
Error: Failed to apply catalog: Command is still failing after 180 seconds expired!
+ rc=1
+ set -e
+ set +ux

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.opendev.org/667815

Changed in tripleo:
assignee: nobody → Jiří Stránský (jistr)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/667821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/667815
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=bac59f433bf538e20b49774080b6299f354e7497
Submitter: Zuul
Branch: master

commit bac59f433bf538e20b49774080b6299f354e7497
Author: Jiri Stransky <email address hidden>
Date: Thu Jun 27 10:57:48 2019 +0200

    Fix rabbitmq staged upgrade

    Fix the short name overriding, and add long name (fqdn) overriding.

    Change-Id: Ia152aed696be15119ba5b75177ef82bc786c4b05
    Partial-Bug: #1832588

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/669699

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/stein)

Reviewed: https://review.opendev.org/669699
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=460ad9ee0a4e58bed8c56ad4cf139063e61be9bc
Submitter: Zuul
Branch: stable/stein

commit 460ad9ee0a4e58bed8c56ad4cf139063e61be9bc
Author: Jiri Stransky <email address hidden>
Date: Thu Jun 27 10:57:48 2019 +0200

    Fix rabbitmq staged upgrade

    Fix the short name overriding, and add long name (fqdn) overriding.

    Change-Id: Ia152aed696be15119ba5b75177ef82bc786c4b05
    Partial-Bug: #1832588
    (cherry picked from commit bac59f433bf538e20b49774080b6299f354e7497)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/670555

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/667821
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=47d1430da57e757f8234d1cce65ad5fee46fa21c
Submitter: Zuul
Branch: master

commit 47d1430da57e757f8234d1cce65ad5fee46fa21c
Author: Jiri Stransky <email address hidden>
Date: Thu Jun 27 11:01:33 2019 +0200

    Upgrade fixes for RabbitMQ and Pacemaker

    Override also RabbitMQ long node names (FQDNs) in addition to the
    short node names we've been already overriding. This allows RabbitMQ
    to start correctly.

    Also remove a superfluous when condition in the pacemaker validation
    task.

    Depends-On: Ia152aed696be15119ba5b75177ef82bc786c4b05
    Change-Id: I9eb274071f3fc2258addd9d2f0c1a5406a702573
    Closes-Bug: #1832588

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.1.0

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/670555
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6e329424f238ae36d582b1010866a5da4837a078
Submitter: Zuul
Branch: stable/stein

commit 6e329424f238ae36d582b1010866a5da4837a078
Author: Jiri Stransky <email address hidden>
Date: Thu Jun 27 11:01:33 2019 +0200

    Upgrade fixes for RabbitMQ and Pacemaker

    Override also RabbitMQ long node names (FQDNs) in addition to the
    short node names we've been already overriding. This allows RabbitMQ
    to start correctly.

    Also remove a superfluous when condition in the pacemaker validation
    task.

    Depends-On: Ia152aed696be15119ba5b75177ef82bc786c4b05
    Change-Id: I9eb274071f3fc2258addd9d2f0c1a5406a702573
    Closes-Bug: #1832588
    (cherry-picked from 47d1430da57e757f8234d1cce65ad5fee46fa21c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.1

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.