kolla

heka elasticsearch is returning warnings

Bug #1560779 reported by Steven Dake
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Steven Dake
kolla newton-1 "n1"
Mitaka
Fix Released
High
Steven Dake
kolla mitaka-rc3 "mitaka-rc3"

Bug Description

with enable_central_logging: yes
246407bf60d3 192.168.1.103:4000/kollaglue/centos-source-heka:3.0.0 "kolla_start" 30 hours ago Up 30 hours heka
[sdake@minime-03 ~]$ docker logs heka
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Creating dest parent directory: /etc/heka
INFO:__main__:Copying /var/lib/kolla/config_files/heka-elasticsearch.toml to /etc/heka/heka-elasticsearch.toml
INFO:__main__:Setting permissions for /etc/heka/heka-elasticsearch.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-global.toml to /etc/heka/heka-global.toml
INFO:__main__:Setting permissions for /etc/heka/heka-global.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-haproxy.toml to /etc/heka/heka-haproxy.toml
INFO:__main__:Setting permissions for /etc/heka/heka-haproxy.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-rabbitmq.toml to /etc/heka/heka-rabbitmq.toml
INFO:__main__:Setting permissions for /etc/heka/heka-rabbitmq.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-openstack.toml to /etc/heka/heka-openstack.toml
INFO:__main__:Setting permissions for /etc/heka/heka-openstack.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-mariadb.toml to /etc/heka/heka-mariadb.toml
INFO:__main__:Setting permissions for /etc/heka/heka-mariadb.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-keystone.toml to /etc/heka/heka-keystone.toml
INFO:__main__:Setting permissions for /etc/heka/heka-keystone.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-auditor.toml to /etc/heka/heka-swift-account-auditor.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-auditor.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-reaper.toml to /etc/heka/heka-swift-account-reaper.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-reaper.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-replicator.toml to /etc/heka/heka-swift-account-replicator.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-replicator.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-server.toml to /etc/heka/heka-swift-account-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-server.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-auditor.toml to /etc/heka/heka-swift-container-auditor.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-auditor.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-replicator.toml to /etc/heka/heka-swift-container-replicator.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-replicator.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-server.toml to /etc/heka/heka-swift-container-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-server.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-updater.toml to /etc/heka/heka-swift-container-updater.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-updater.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-object-auditor.toml to /etc/heka/heka-swift-object-auditor.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-object-auditor.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-object-expirer.toml to /etc/heka/heka-swift-object-expirer.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-object-expirer.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-object-replicator.toml to /etc/heka/heka-swift-object-replicator.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-object-replicator.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-object-server.toml to /etc/heka/heka-swift-object-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-object-server.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-object-updater.toml to /etc/heka/heka-swift-object-updater.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-object-updater.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-proxy-server.toml to /etc/heka/heka-swift-proxy-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-proxy-server.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-rsyncd.toml to /etc/heka/heka-swift-rsyncd.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-rsyncd.toml
INFO:__main__:Writing out command to execute
Running command: '/usr/bin/hekad -config=/etc/heka/'
2016/03/21 20:33:49 Pre-loading: [elasticsearch_json_encoder]
2016/03/21 20:33:49 Pre-loading: [elasticsearch_output]
2016/03/21 20:33:49 Pre-loading: [syslog_encoder]
2016/03/21 20:33:49 Pre-loading: [syslog_log_decoder]
2016/03/21 20:33:49 Pre-loading: [swift_log_decoder]
2016/03/21 20:33:49 Pre-loading: [multi_log_decoder]
2016/03/21 20:33:49 Pre-loading: [syslog_input]
2016/03/21 20:33:49 Pre-loading: [haproxy_file_output]
2016/03/21 20:33:49 Pre-loading: [keepalived_file_output]
2016/03/21 20:33:49 Pre-loading: [keystone_apache_log_decoder]
2016/03/21 20:33:49 Pre-loading: [keystone_apache_logstreamer_input]
2016/03/21 20:33:49 Pre-loading: [mariadb_log_decoder]
2016/03/21 20:33:49 Pre-loading: [mariadb_logstreamer_input]
2016/03/21 20:33:49 Pre-loading: [openstack_log_decoder]
2016/03/21 20:33:49 Pre-loading: [openstack_logstreamer_input]
2016/03/21 20:33:49 Pre-loading: [rabbitmq_log_decoder]
2016/03/21 20:33:49 Pre-loading: [rabbitmq_log_splitter]
2016/03/21 20:33:49 Pre-loading: [rabbitmq_logstreamer_input]
2016/03/21 20:33:49 Pre-loading: [swift-account-auditor_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-account-reaper_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-account-replicator_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-account-server_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-container-auditor_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-container-replicator_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-container-server_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-container-updater_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-object-auditor_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-object-expirer_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-object-replicator_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-object-server_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-object-updater_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-proxy-server_file_output]
2016/03/21 20:33:49 Pre-loading: [swift-rsyncd_file_output]
2016/03/21 20:33:49 Pre-loading: [ProtobufDecoder]
2016/03/21 20:33:49 Loading: [ProtobufDecoder]
2016/03/21 20:33:49 Pre-loading: [ProtobufEncoder]
2016/03/21 20:33:49 Loading: [ProtobufEncoder]
2016/03/21 20:33:49 Pre-loading: [TokenSplitter]
2016/03/21 20:33:49 Loading: [TokenSplitter]
2016/03/21 20:33:49 Pre-loading: [HekaFramingSplitter]
2016/03/21 20:33:49 Loading: [HekaFramingSplitter]
2016/03/21 20:33:49 Pre-loading: [NullSplitter]
2016/03/21 20:33:49 Loading: [NullSplitter]
2016/03/21 20:33:49 Loading: [syslog_log_decoder]
2016/03/21 20:33:49 Loading: [swift_log_decoder]
2016/03/21 20:33:49 Loading: [keystone_apache_log_decoder]
2016/03/21 20:33:49 Loading: [mariadb_log_decoder]
2016/03/21 20:33:49 Loading: [openstack_log_decoder]
2016/03/21 20:33:49 Loading: [rabbitmq_log_decoder]
2016/03/21 20:33:49 Loading: [multi_log_decoder]
2016/03/21 20:33:49 Loading: [elasticsearch_json_encoder]
2016/03/21 20:33:49 Loading: [syslog_encoder]
2016/03/21 20:33:49 Loading: [rabbitmq_log_splitter]
2016/03/21 20:33:49 Loading: [syslog_input]
2016/03/21 20:33:49 Loading: [keystone_apache_logstreamer_input]
2016/03/21 20:33:49 Loading: [mariadb_logstreamer_input]
2016/03/21 20:33:49 Loading: [openstack_logstreamer_input]
2016/03/21 20:33:49 Loading: [rabbitmq_logstreamer_input]
2016/03/21 20:33:49 Loading: [elasticsearch_output]
2016/03/21 20:33:49 Loading: [haproxy_file_output]
2016/03/21 20:33:49 Loading: [keepalived_file_output]
2016/03/21 20:33:49 Loading: [swift-account-auditor_file_output]
2016/03/21 20:33:49 Loading: [swift-account-reaper_file_output]
2016/03/21 20:33:49 Loading: [swift-account-replicator_file_output]
2016/03/21 20:33:49 Loading: [swift-account-server_file_output]
2016/03/21 20:33:49 Loading: [swift-container-auditor_file_output]
2016/03/21 20:33:49 Loading: [swift-container-replicator_file_output]
2016/03/21 20:33:49 Loading: [swift-container-server_file_output]
2016/03/21 20:33:49 Loading: [swift-container-updater_file_output]
2016/03/21 20:33:49 Loading: [swift-object-auditor_file_output]
2016/03/21 20:33:49 Loading: [swift-object-expirer_file_output]
2016/03/21 20:33:49 Loading: [swift-object-replicator_file_output]
2016/03/21 20:33:49 Loading: [swift-object-server_file_output]
2016/03/21 20:33:49 Loading: [swift-object-updater_file_output]
2016/03/21 20:33:49 Loading: [swift-proxy-server_file_output]
2016/03/21 20:33:49 Loading: [swift-rsyncd_file_output]
2016/03/21 20:33:49 Starting hekad...
2016/03/21 20:33:49 Output started: swift-container-replicator_file_output
2016/03/21 20:33:49 Output started: swift-object-auditor_file_output
2016/03/21 20:33:49 Output started: swift-object-updater_file_output
2016/03/21 20:33:49 Output started: swift-proxy-server_file_output
2016/03/21 20:33:49 Output started: swift-rsyncd_file_output
2016/03/21 20:33:49 Output started: keepalived_file_output
2016/03/21 20:33:49 Output started: swift-container-auditor_file_output
2016/03/21 20:33:49 Output started: swift-account-server_file_output
2016/03/21 20:33:49 Output started: swift-object-replicator_file_output
2016/03/21 20:33:49 Output started: haproxy_file_output
2016/03/21 20:33:49 Output started: swift-account-replicator_file_output
2016/03/21 20:33:49 Output started: swift-container-updater_file_output
2016/03/21 20:33:49 Output started: swift-object-expirer_file_output
2016/03/21 20:33:49 Output started: swift-account-auditor_file_output
2016/03/21 20:33:49 Output started: swift-container-server_file_output
2016/03/21 20:33:49 Output started: swift-object-server_file_output
2016/03/21 20:33:49 Output started: elasticsearch_output
2016/03/21 20:33:49 Output started: swift-account-reaper_file_output
2016/03/21 20:33:49 MessageRouter started.
2016/03/21 20:33:49 Input started: rabbitmq_logstreamer_input
2016/03/21 20:33:49 Input started: syslog_input
2016/03/21 20:33:49 Input started: keystone_apache_logstreamer_input
2016/03/21 20:33:49 Input started: mariadb_logstreamer_input
2016/03/21 20:33:49 Input started: openstack_logstreamer_input
2016/03/21 20:34:06 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host
2016/03/21 20:34:09 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host
2016/03/21 20:34:11 Plugin 'elasticsearch_output' error: can't index: HTTP response didn't contain valid JSON. Status: 503 Service Unavailable. Body: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
2016/03/21 20:34:13 Plugin 'elasticsearch_output' error: can't index: HTTP response didn't contain valid JSON. Status: 503 Service Unavailable. Body: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
2016/03/21 20:34:18 Plugin 'elasticsearch_output' error: can't index: HTTP response didn't contain valid JSON. Status: 503 Service Unavailable. Body: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
2016/03/21 20:34:23 Plugin 'elasticsearch_output' error: can't index: HTTP response didn't contain valid JSON. Status: 503 Service Unavailable. Body: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
2016/03/21 20:34:28 Plugin 'elasticsearch_output' error: can't index: HTTP response didn't contain valid JSON. Status: 503 Service Unavailable. Body: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Steven Dake (sdake)
Changed in kolla:
status: New → Confirmed
importance: Undecided → High
milestone: none → newton-1
summary: - heka elasticserach is busted
+ heka elasticsearch is busted
Vikram Hosakote (vhosakot)
Changed in kolla:
assignee: nobody → Vikram Hosakote (vhosakot)
Revision history for this message
Steven Dake (sdake) wrote : Re: heka elasticsearch is busted

Eric has stated this is a warning triggered by the fact that elasticsearch isn't enabled. enabling centrla logging may fix the problem. None the less, we should remove warnings like this in our releaes of mitaka because they make things look busted.

I think a conditional on copying of the elasticsearch fiel should get the job done.

Changed in kolla:
importance: High → Medium
assignee: Vikram Hosakote (vhosakot) → Steven Dake (sdake)
summary: - heka elasticsearch is busted
+ heka elasticsearch is returning warnings
Revision history for this message
Steven Dake (sdake) wrote :
Download full text (12.6 KiB)

These are the logs when enable_central_logging is enabled:

[sdake@minime-03 kolla]$ docker logs heka
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Creating dest parent directory: /etc/heka
INFO:__main__:Copying /var/lib/kolla/config_files/heka-elasticsearch.toml to /etc/heka/heka-elasticsearch.toml
INFO:__main__:Setting permissions for /etc/heka/heka-elasticsearch.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-global.toml to /etc/heka/heka-global.toml
INFO:__main__:Setting permissions for /etc/heka/heka-global.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-haproxy.toml to /etc/heka/heka-haproxy.toml
INFO:__main__:Setting permissions for /etc/heka/heka-haproxy.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-rabbitmq.toml to /etc/heka/heka-rabbitmq.toml
INFO:__main__:Setting permissions for /etc/heka/heka-rabbitmq.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-openstack.toml to /etc/heka/heka-openstack.toml
INFO:__main__:Setting permissions for /etc/heka/heka-openstack.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-mariadb.toml to /etc/heka/heka-mariadb.toml
INFO:__main__:Setting permissions for /etc/heka/heka-mariadb.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-keystone.toml to /etc/heka/heka-keystone.toml
INFO:__main__:Setting permissions for /etc/heka/heka-keystone.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-horizon.toml to /etc/heka/heka-horizon.toml
INFO:__main__:Setting permissions for /etc/heka/heka-horizon.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-auditor.toml to /etc/heka/heka-swift-account-auditor.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-auditor.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-reaper.toml to /etc/heka/heka-swift-account-reaper.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-reaper.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-replicator.toml to /etc/heka/heka-swift-account-replicator.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-replicator.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-account-server.toml to /etc/heka/heka-swift-account-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-account-server.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-auditor.toml to /etc/heka/heka-swift-container-auditor.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-auditor.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-replicator.toml to /etc/heka/heka-swift-container-replicator.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-replicator.toml
INFO:__main__:Copying /var/lib/kolla/config_files/heka-swift-container-server.toml to /etc/heka/heka-swift-container-server.toml
INFO:__main__:Setting permissions for /etc/heka/heka-swift-container-server.toml...

Revision history for this message
Steven Dake (sdake) wrote :

Instead of those logs, we get these with elasticsearch starting prior to haproxy and keepalived.
2016/03/23 22:12:19 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host
2016/03/23 22:12:22 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host
2016/03/23 22:12:25 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host
2016/03/23 22:12:28 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://192.168.1.148:9200/_bulk: dial tcp 192.168.1.148:9200: no route to host

OpenStack Infra (hudson-openstack)
Changed in kolla:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/296818
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=0d2591dc30ac4d7c53b03d77006623a2162509fd
Submitter: Jenkins
Branch: master

commit 0d2591dc30ac4d7c53b03d77006623a2162509fd
Author: Steven Dake <email address hidden>
Date: Wed Mar 23 18:16:04 2016 -0400

    Reduce log errors from heka

    Heka depends on haproxy and keepaived being present to communicate
    with ElasticSearch. If we start ElasticSearch prior to haproxy and
    keepalived, the number of errors are reduced in heka.

    Change-Id: Id2c742ea572c6450a371421e21f34aa69355bb8b
    Partial-Bug: #1560779

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/300662

Steven Dake (sdake)
Changed in kolla:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/300662
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=62cb45e7208342fcb6727f979ffa2c80126005a7
Submitter: Jenkins
Branch: stable/mitaka

commit 62cb45e7208342fcb6727f979ffa2c80126005a7
Author: Steven Dake <email address hidden>
Date: Wed Mar 23 18:16:04 2016 -0400

    Reduce log errors from heka

    Heka depends on haproxy and keepaived being present to communicate
    with ElasticSearch. If we start ElasticSearch prior to haproxy and
    keepalived, the number of errors are reduced in heka.

    Change-Id: Id2c742ea572c6450a371421e21f34aa69355bb8b
    Partial-Bug: #1560779
    (cherry picked from commit 0d2591dc30ac4d7c53b03d77006623a2162509fd)

tags: added: in-stable-mitaka
Swapnil Kulkarni (coolsvap-deactivatedaccount)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.