incompatible marshal file format (can't be read) format version 4.8 required; 89.111 given

[Anastasia Palkina]

* AFFECTS 4.0 ONLY *

ISO #55
"release": "4.0",
"nailgun_sha": "d54c4e53658318db4d58ba89e2aa1cf97b0de5b2",
"ostf_sha": "59862f01522eb1b9b44eac82d7c21e425f143440",
"astute_sha": "df6ddea3abc93fbe1cab9b4534d4d5e9508c95d6",
"fuellib_sha": "ac52f3cf485bde0e445fb987eb31aba367dd3fb3"

1. Create new environment (CentOS or Ubuntu)
2. Add controller and compute nodes
3. Start deployment
4. Progress of provisioning is 100 but all hangs, deployment doesn't started

[root@fuel ~]# mco ping -v
The ping application failed to run: incompatible marshal file format (can't be read)
 format version 4.8 required; 89.111 given

And there is error in mcollective.log (see screen)

[root@fuel ~]# grep "err:" /var/log/puppet/bootstrap_admin_node.log
err: /Stage[main]/Mcollective::Rabbitmq/Exec[create-mcollective-broadcast-exchange]/returns: change from notrun to 0 failed: curl -i -u mcollective:marionette -H "content-type:application/json" -XPUT -d'{"type":"topic","durable":true}' http://localhost:55672/api/exchanges/mcollective/mcollective_broadcast returned 7 instead of one of [0] at /etc/puppet/modules/mcollective/manifests/rabbitmq.pp:118
err: /Stage[main]/Mcollective::Rabbitmq/Exec[create-mcollective-directed-exchange]/returns: change from notrun to 0 failed: curl -i -u mcollective:marionette -H "content-type:application/json" -XPUT -d'{"type":"direct","durable":true}' http://localhost:55672/api/exchanges/mcollective/mcollective_directed returned 7 instead of one of [0] at /etc/puppet/modules/mcollective/manifests/rabbitmq.pp:110

[Mike Scherbakov (mihgen)]

I remember we were working on this bug. Though I don't remember what was the cause... Possible reasons:
- your configs differ, and client/server has direct_addressing=1 and server/client =0
- you try to run mcollective in multithreaded fashion

[Vladimir Sharshov (vsharshov)]

Vladimir, why for 3.2.1? At now moment this affect only 4.0 because only where we publish new stomp packages.

[Vladimir Sharshov (vsharshov)]

Get this bug today first time on 3.2.1 with old stomp gem (1.2.8) instead of new 1.2.16 in 4.0 which i thought the main reason for such a problem: problem with old mcollective and new stomp.

{"release": "3.2.1", "nailgun_sha": "68b064d77d951f20fd33f11de8976d113df52fc5", "ostf_sha": "736a0e2f88fb5b5bdd1c3117d5acb942d428e747", "astute_sha": "c981ba40fccfdb78be85fcff13188dfded2773ee", "fuellib_sha": "748c00d079753242f9b087a9ee82bc7fde53bbb5"}

[Vladimir Sharshov (vsharshov)]

This error also go with message "ERROR -- : rabbitmq.rb:30:in `on_miscerr' Unexpected error on connection stomp://mcollective@10.20.0.2:61613: es_recv: connection.receive returning EOF as nil - resetting connection." in cluster nodes and error in RabbitMQ log:

=ERROR REPORT==== 3-Dec-2013::04:10:01 ===
STOMP error frame sent:
Message: not_found
Detail: "NOT_FOUND - no exchange 'mcollective_broadcast' in vhost 'mcollective'\n"
Server private detail: none

This problem was described and solved as i can see in https://bugs.launchpad.net/fuel/+bug/1252269. We should backport this fix also for 3.2.1

[Vladimir Sharshov (vsharshov)]

Update: fix already in https://github.com/Mirantis/fuel/tree/3.2-fixes branch and this error was reproduced with this fix. Fix do not help(

Fix commit: https://github.com/Mirantis/fuel/commit/748c00d079753242f9b087a9ee82bc7fde53bbb5
"fuellib_sha": "748c00d079753242f9b087a9ee82bc7fde53bbb5"

[Vladimir Sharshov (vsharshov)]

Now, i think, we can mark this bug as duplicate and update description of https://bugs.launchpad.net/fuel/+bug/1252269, because it describe different evidences of same problem.

More data:
ISO 3.2.1 #84
No error in bootstrap_admin_node.log in this time.

[root@fuel ~]# grep "err:" /var/log/puppet/bootstrap_admin_node.log
[root@fuel ~]#

I save env for Dmitry Ilyin

[Aleksandr Shaposhnikov (alashai8)]

I'm not sure that bug #1252269 is the same problem. Basically I have this problem right now on 4.0 build 92.