compute host "has not been heard from in a while" failing ceph jobs

[Matt Riedemann]

Seen here:

http://logs.openstack.org/89/283689/3/gate/gate-tempest-dsvm-full-ceph-src-glance_store/f23f9eb/logs/screen-n-sch.txt.gz?level=TRACE

and here:

http://logs.openstack.org/08/283708/3/check/gate-tempest-dsvm-full-ceph/094fc27/logs/screen-n-sch.txt.gz?level=TRACE

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22has%20not%20been%20heard%20from%20in%20a%20while%5C%22%20AND%20tags%3A%5C%22screen-n-sch.txt%5C%22%20AND%20build_name%3A*ceph*&from=7d

[Matt Riedemann (mriedem)]

According to logstash this shows up on the OVH and Internap nodes, which are the slowest ones we have.

[Matt Riedemann (mriedem)]

Updated query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22has%20not%20been%20heard%20from%20in%20a%20while%5C%22%20AND%20tags%3A%5C%22screen-n-sch.txt%5C%22%20AND%20(build_name%3A%5C%22gate-tempest-dsvm-full-ceph%5C%22%20OR%20build_name%3A%5C%22gate-tempest-dsvm-full-ceph-src-os-brick%5C%22%20OR%20build_name%3A%5C%22gate-tempest-dsvm-full-ceph-src-glance_store%5C%22)&from=7d

[Matt Riedemann (mriedem)]

Also hitting on RAX nodes now.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/284980

[Matt Riedemann (mriedem)]

idle cpu takes a big hit starting around 25-02 01:41:04 here:

http://logs.openstack.org/89/283689/3/gate/gate-tempest-dsvm-full-ceph-src-glance_store/f23f9eb/logs/screen-dstat.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/284980
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d1cc60a21559e6065b41b1ecbd73df12bab77b6a
Submitter: Jenkins
Branch: master

commit d1cc60a21559e6065b41b1ecbd73df12bab77b6a
Author: Matt Riedemann <email address hidden>
Date: Thu Feb 25 18:21:54 2016 -0500

    Add service binary/host to service is down log for context

    When we're logging that a service appears to be down in the
    DB service group drive, add the service binary and host to
    the message for context.

    Change-Id: I8f676f7b498f4d7cb9b574a401ec892c2a3aacd3
    Related-Bug: #1549854