RabbitMQ connections lack heartbeat or TCP keepalives

[Dmitry Mescheryakov]

This is the copy of bug https://bugs.launchpad.net/nova/+bug/856764

[Bogdan Dobrelya (bogdando)]

related bug https://bugs.launchpad.net/oslo.messaging/+bug/1338732

[Igor Marnat (imarnat)]

Patches and requests for building packages: https://bugs.launchpad.net/mos/+bug/1340711

[Serg Melikyan (smelikyan)]

Fixed by https://gerrit.mirantis.com/19112

[Dmitry Mescheryakov (dmitrymex)]

Here is the summary changes for 5.0.1:
https://gerrit.mirantis.com/19900
https://gerrit.mirantis.com/19112
https://gerrit.mirantis.com/19573
https://review.openstack.org/112681

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20691/centos

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

[Dmitry Mescheryakov (dmitrymex)]

Porting of changes from 5.0.1 to 5.1 is done:

https://gerrit.mirantis.com/19902
https://gerrit.mirantis.com/19906
https://gerrit.mirantis.com/19905
https://review.openstack.org/112677

[Serg Melikyan (smelikyan)]

https://gerrit.mirantis.com/20691 - Proper heartbeat implementation is submitted and ready to be tested

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/21233
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21233/centos

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/21233
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-21233/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-20691/centos

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-20691/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.messaging has been built from changeset: http://gerrit.mirantis.com/20691
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos

[Dmitry Borodaenko (angdraug)]

What is the current status of this bug in 5.1 and 5.0.2?

[Serg Melikyan (smelikyan)]

We verified and merged fix to 5.1 branch, backports to 5.0.2 are pending:
https://review.openstack.org/117500
http://gerrit.mirantis.com/21389

[Serg Melikyan (smelikyan)]

Backport to 5.0.2 merged

[Andrey Sledzinskiy (asledzinskiy)]

{

    "build_id": "2014-09-15_00-01-46",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "8",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8",
    "production": "docker",
    "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b",
    "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
    "feature_groups": [
        "experimental"
    ],
    "release": "5.1",
    "release_versions": {
        "2014.1.1-5.1": {
            "VERSION": {
                "build_id": "2014-09-15_00-01-46",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "8",
                "api": "1.0",
                "nailgun_sha": "b8d8189cc37d6d1b26f4479be6be7313beefb1c8",
                "production": "docker",
                "fuelmain_sha": "d7ed7973034bde73d3f42c000984423b59b2312b",
                "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
                "feature_groups": [
                    "experimental"
                ],
                "release": "5.1",
                "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"
            }
        }
    },
    "fuellib_sha": "395fd9d20a003603cc9ad26e16cb13c1c45e24e6"

}

We have problems with oslo.messaging in our tests after restart nodes
Steps:
1. Create next cluster - Ubuntu, Simple, Flat nova-network, Cinder for volumes, Ceph for images, 1 controller, 1 compute, 2 cinder+ceph nodes
2. Deploy cluster
3. Reboot all nodes one by one
4. Open Health Check tab
5. Run tests

Expected - tests passed
Actual - tests on volume and instance creation failed. Lots of errors on compute node in nova logs:
2014-09-15 13:08:11 ERROR

oslo.messaging._drivers.impl_rabbit [req-c4c51ee4-f9b7-4e9e-a949-e694f3b21757 ] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.

And on cinder node in cinder logs
2014-09-15 07:40:19 ERROR

oslo.messaging._drivers.impl_rabbit [req-2fd99d10-2de0-478e-ab3e-cabc52935455 - - - - -] AMQP server on 10.108.52.4:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.

Logs are attached

[Andrey Sledzinskiy (asledzinskiy)]

Also same errors in next configuration - Ubuntu, Ha, Flat Nova network, Cinder for volumes, 3 controllers, 2 compute
After destroy of one of controllers some OSTF tests failed with timeout
Errors on node-4 (compute)
nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Timed out waiting for a reply to message ID 2dd0eabc69984cdca76067420d9d6f12

Logs are attached

[Serg Melikyan (smelikyan)]

This exception has nothing to do with heartbeats, from log there is clearly issue with connection to rabbitmq. Looks like server are down.

[Serg Melikyan (smelikyan)]

I suggest opening new bug and returning state of this bug to commited

[Andrey Sledzinskiy (asledzinskiy)]

I agree with it

[Bogdan Dobrelya (bogdando)]

related bug related bug https://bugs.launchpad.net/mos/+bug/1371723