Mcollective has broken: rescue in block in run' /usr/libexec/mcollective/mcollective/security/psk.rb:22:in `load'

[Anastasia Palkina]

"build_id": "2014-04-23_03-41-25",
"mirantis": "yes",
"build_number": "127",
"nailgun_sha": "ead54af79fb8d798bdf8b7cfc08ee60103c9292f",
"production": "prod",
"ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b",
"fuelmain_sha": "a864e28fff841b40c3b4a53f00e622002f300b19",
"astute_sha": "6e8fa4cc12968d7b468fc590b2f06bb59bf74511",
"release": "5.0",
"fuellib_sha": "646e52b7a34a0027b43c46aa5b525fe7674a8059"

Diagnostic snapshot doesn't work.

In /var/log/mcollective.log:

W, [2014-04-23T08:15:18.119478 #2522] WARN -- : runner.rb:78:in `rescue in block in run' Failed to handle message: incompatible marshal file format (can't be read)
        format version 4.8 required; 89.111 given - TypeError

W, [2014-04-23T08:15:18.119628 #2522] WARN -- : runner.rb:79:in `rescue in block in run' /usr/libexec/mcollective/mcollective/security/psk.rb:22:in `load'
        /usr/libexec/mcollective/mcollective/security/psk.rb:22:in `decodemsg'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/message.rb:178:in `decode!'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/runner.rb:133:in `receive'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/runner.rb:59:in `block in run'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/runner.rb:57:in `loop'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/runner.rb:57:in `run'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/unix_daemon.rb:30:in `block in daemonize_runner'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/unix_daemon.rb:13:in `block in daemonize'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/unix_daemon.rb:5:in `fork'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/unix_daemon.rb:5:in `daemonize'
        /usr/lib64/ruby/site_ruby/2.1.0/mcollective/unix_daemon.rb:20:in `daemonize_runner'
        /usr/sbin/mcollectived:47:in `<main>'

[Anastasia Palkina (apalkina)]

[root@fuel ~]# bootstrap_admin_node.sh

Press a key to enter Fuel Setup (or press ESC to skip)... 15
Skipping Fuel Setup..
Applying default Fuel setings...Done!
Notice: Compiled catalog for fuel.domain.tld in environment production in 5.76 seconds
Notice: /Stage[main]/Openstack::Logrotate/Exec[logrotate_check]/returns: executed successfully
Notice: /Stage[main]/Mcollective::Server/File[/etc/mcollective/server.cfg]/content: content changed '{md5}0fff75a0d2e9f918219dff4ce235f2a1' to '{md5}c50b402633cbed42fdad4c6d5072293b'
Notice: /Stage[main]/Nailgun::Ostf/Exec[ostf-init]/returns: executed successfully
Notice: /Stage[main]/Rabbitmq::Server/Stdlib::Safe_package[qpid-cpp-server]/Package[qpid-cpp-server]/ensure: created
Notice: Module rabbitmq cannot notify service rabbitmq-server on package rabbitmq-server update
Notice: /Stage[main]/Rabbitmq::Service/Notify[Module rabbitmq cannot notify service rabbitmq-server on package rabbitmq-server update]/message: defined 'message' as 'Module rabbitmq cannot notify service rabbitmq-server on package rabbitmq-server update'
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: % Total % Received % Xferd Average Speed Time Time Time Current
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: Dload Upload Total Spent Left Speed
  0 0 0 0 0 31 0 5475 --:--:-- --:--:-- --:--:-- 0nge]/returns:
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: HTTP/1.1 204 No Content
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: Date: Wed, 23 Apr 2014 08:59:04 GMT
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: Content-Type: application/json
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: Content-Length: 0
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns:
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: executed successfully
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: % Total % Received % Xferd Average Speed Time Time Time Current
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: Dload Upload Total Spent Left Speed
  0 0 0 0 0 32 0 5847 --:--:-- --:--:-- --:--:-- 0ge]/returns:
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: HTTP/1.1 204 No Content
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Notice: /Stage[main]/Nailgun::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: Date: Wed, 23 Apr 2014 08:59:04 GMT
Notice: /Stage[...

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/89823

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/89823
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f44ea3885ca11c38a63ce51aac6f0919a6ea4186
Submitter: Jenkins
Branch: master

commit f44ea3885ca11c38a63ce51aac6f0919a6ea4186
Author: Vladimir Sharshov <email address hidden>
Date: Wed Apr 23 14:05:04 2014 +0400

    Setup RabbitMQ before mcollective installation

    * install RabbitMQ and create mcollective exchanges;
    * restart mcollective service if config is changed;
    * hearbeat for master node.

    Thanks Dmitry Iliyn for helps.

    Change-Id: I7b9006510d7fa747b3edf37ad724063cef122f3f
    Closes-Bug: #1311530

[Anastasia Palkina (apalkina)]

Verified on ISO #133
"build_id": "2014-04-24_01-00-26",
"mirantis": "yes",
"build_number": "133",
"nailgun_sha": "3e955a06aab9baf03210fef5d606971ef462be6c",
"production": "prod",
"ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b",
"fuelmain_sha": "387d2e931ee14d5773f21f1e2860a13b50b37d04",
"astute_sha": "6e8fa4cc12968d7b468fc590b2f06bb59bf74511",
"release": "5.0",
"fuellib_sha": "b9b8c1a951d777ece3b4dc400644ff1167f4fe01"