javelin2 ceilometer create failure: AssertionError: expecting at least one sample

[Joe Gordon]

http://logs.openstack.org/23/111223/23/gate/gate-grenade-dsvm/ed5c23a/logs/grenade.sh.txt.gz

http://logs.openstack.org/23/111223/23/gate/gate-grenade-dsvm/ed5c23a/logs/javelin.txt.gz

2014-09-06 19:24:21.354 29125 CRITICAL tempest [-] AssertionError: expecting at least one sample
2014-09-06 19:24:21.354 29125 TRACE tempest Traceback (most recent call last):
2014-09-06 19:24:21.354 29125 TRACE tempest File "/usr/local/bin/javelin2", line 10, in
2014-09-06 19:24:21.354 29125 TRACE tempest sys.exit(main())
2014-09-06 19:24:21.354 29125 TRACE tempest File "/opt/stack/old/tempest/tempest/cmd/javelin.py", line 622, in main
2014-09-06 19:24:21.354 29125 TRACE tempest checker.check()
2014-09-06 19:24:21.354 29125 TRACE tempest File "/opt/stack/old/tempest/tempest/cmd/javelin.py", line 205, in check
2014-09-06 19:24:21.354 29125 TRACE tempest self.check_telemetry()
2014-09-06 19:24:21.354 29125 TRACE tempest File "/opt/stack/old/tempest/tempest/cmd/javelin.py", line 279, in check_telemetry
2014-09-06 19:24:21.354 29125 TRACE tempest self.assertTrue(len(body) >= 1, 'expecting at least one sample')
2014-09-06 19:24:21.354 29125 TRACE tempest File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
2014-09-06 19:24:21.354 29125 TRACE tempest raise self.failureException(msg)
2014-09-06 19:24:21.354 29125 TRACE tempest AssertionError: expecting at least one sample
2014-09-06 19:24:21.354 29125 TRACE tempest

Note: until https://review.openstack.org/#/c/113663/ lands we cannot add a fingerprint for this bug into elastic-recheck to track its frequency etc.

[Chris Dent (cdent)]

I suspect this is a latency problem. I've seen evidence elsewhere in the gate that it can take up to 120 seconds for a notification to make the journey from its origin, traverse the message bus and then land in the database. I'm not sure where it's doing during that time, malingering for sure.

[Chris Dent (cdent)]

Update:

Sigh, actually with more digging it appears the notification agent isn't running at all thus no samples will be caught, have a look at http://logs.openstack.org/23/111223/23/gate/gate-grenade-dsvm/ed5c23a/logs/ps.txt.gz

There's also no log file. However the log for grenade.sh indicates it is supposed to be started, but there are no artifacts that I've found yet as to why it isn't present.

Will keep digging, but anybody have ideas or history on this?

[David Kranz (david-kranz)]

Sounds like this is not a tempest issue. Please reopen if new evidence is discovered.

[gordon chung (chungg)]

cdent. is it correct that this seems like a random error.... notification agent seems to be running for all the recent patches i've seen.

setting low for now.

[Chris Dent (cdent)]

After some digging and discussion with the qa crew, there turns out to be an intermittent problem in the gate on some hosts where services will randomly not start and because of the use of screen the details of the how and why are being lost.

This change will move things forward: https://review.openstack.org/#/c/120077/

So the problem is at the level of infrastructure, not something specific to either tempest or ceilometer but impacting both.

[gordon chung (chungg)]

marking invalid for ceilometer.