The e-r graph is failing to build because we apparently have some timestamps that are too large, I'm not sure what that means, if they are indexes or what, but you can see this when running the uncategorized_fails or graph CLI:
kedExecutionException[org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]; nested: CircuitBreakingException[[FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]; ', u'index': u'logstash-2016.10.03', u'status': 500, u'shard': 4}, {u'reason': u'QueryPhaseExecutionException[[logstash-2016.10.04][4]: query[+((message:"bash install_package command not found") (+(message:"ssh connect to host") +(message:"connection timed out")) (message:"error ssh encountered an unknown error during the connection") (+(message:"error opt stack new grenade projects 70_cinder resources.sh") +(message:"ssh to the client did not work something very wrong"))) +tags:console +voting:1],from[0],size[30000],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@200428e8>!]: Query Failed [Failed to execute main query]]; nested: ElasticsearchException[org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]; nested: UncheckedExecutionException[org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]; nested: CircuitBreakingException[[FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]; ', u'index': u'logstash-2016.10.04', u'status': 500, u'shard': 4}]}, u'timed_out': False, u'took': 4709}
Traceback (most recent call last):
File ".tox/venv/bin/elastic-recheck-uncategorized", line 10, in <module>
sys.exit(main())
File "/home/user/git/elastic-recheck/elastic_recheck/cmd/uncategorized_fails.py", line 361, in main
data = collect_metrics(classifier, fails, config=config)
File "/home/user/git/elastic-recheck/elastic_recheck/cmd/uncategorized_fails.py", line 314, in collect_metrics
hits = _status_count(results)
File "/home/user/git/elastic-recheck/elastic_recheck/cmd/uncategorized_fails.py", line 256, in _status_count
["build_status", "build_uuid"])
File "/home/user/git/elastic-recheck/elastic_recheck/results.py", line 171, in detect_facets
attr = self._histogram(hit[facet], facet)
File "/home/user/git/elastic-recheck/elastic_recheck/results.py", line 196, in __getitem__
return self.__getattr__(key)
File "/home/user/git/elastic-recheck/elastic_recheck/results.py", line 221, in __getattr__
result = first(self._hit['_source'][attr])
File "/home/user/git/elastic-recheck/elastic_recheck/results.py", line 215, in first
return item[0]
IndexError: list index out of range
https:/