Comment 14 for bug 1627106

Revision history for this message
IWAMOTO Toshihiro (iwamoto) wrote : Re: TimeoutException while executing test_post_commit_vswitchd_completed_no_failures

This comment is about "no response to inactivity probe after 5 seconds, disconnecting" messages from ovsdb-server and ovs-vswitchd. I believe this is from the same cause as this bug report, but it can turn out to be otherwise.

1. Query the logstash with message:"no response to inactivity probe". Use tcpdump or whatsoever to get the json.

import simplejson
import collections
for k,v in collections.Counter([x['_source']['log_url'] for x in j['hits']['hits']]).most_common():
    print('%d\t%s' % (v,k))

gives something like:


Note the issue is not limited to neutron or tempest. functional and fullstack are also on the list, but not useful for this analysis as they don't leave dstat logs.

2. Get syslog.txt and dstat-csv_log.txt.
3. Compare them. for example:

$ grep 'no response to inacti' 426842/syslog.txt.gz |colrm 80
$ python < 426842/dstat-csv_log.txt.gz |colrm 80
$ cat
import calendar
import time
import sys

yr = time.gmtime().tm_year
told = None
thres = 3 # there is some room for tuning :)
for l in sys.stdin:
    l = l[:-1]
    ll = l.split(',')
        tm = time.strptime(ll[0], "%d-%m %H:%M:%S")
        tnew = calendar.timegm(time.struct_time((yr,) + tm[1:]))
        if told and tnew - told > thres:
            print(tnew - told)
        told = tnew
    except ValueError:

Their timestamps are not identical, but highly correlated imo.