I followed option three to create a logging capability. I was testing this against my charms, for a storm cluster. Essentially I want a 2 node elastic search cluster, an indexer, Kibana and agents on all my nodes.
I am running the juju client on OSX 10.8.2, against AWS.
The commands that I followed are as follows (after deploying and verifying the storm cluster nodes):
juju deploy elasticsearch
juju add-unit elasticsearch
juju deploy logstash-indexer
juju add-relation logstash-indexer elasticsearch:cluster
juju deploy kibana
juju add-relation kibana elasticsearch:rest
juju expose kibana
juju deploy logstash-agent
juju add-relation logstash-agent logstash-indexer:input
juju add-relation logstash-agent stormworker
juju add-relation logstash-agent stormmaster
juju set logstash-agent CustomLogFile="['/var/log/*.log','/opt/storm/storm-0.8.1/logs/*.*']"
juju set logstash-agent Reload="Other"
The error that I then get on the Kibana interface is:
Oops! Something went terribly wrong.
I'm not totally sure what happened, but maybe refreshing, or hitting Reset will help. If that doesn't work, you can try restarting your browser. If all else fails, it is possible your configuation has something funky going on.
If it helps, I received a 500 Internal Server Error from: api/search/eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjkwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzNzc3NzgxMjM1OTN9?_=1377778123596
These are the Juju debug logs relating to the log-stash agent:
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju uniter.go:350 worker/uniter: running "config-changed" hook
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK + HOME=/var/lib/juju/agents/unit-logstash-agent-1/charm
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK + . /var/lib/juju/agents/unit-logstash-agent-1/charm/hooks/common.sh
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ VERSION=1.1.4
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ BASEURL=https://logstash.objects.dreamhost.com/release
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ CHECKSUM=68ac6c953aad026752f69ae7c15b13f4
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ BASEPATH=/opt/logstash-agent
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ USER=logstash
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK +++ unit-get private-address
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "unit-get" ["private-address"]
ip-10-151-65-55:2013-08-29 12:02:37 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ HOST=ip-10-151-65-55.ec2.internal
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK ++ juju-log 'Finished loading config variables'
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Finished loading config variables"]
ip-10-151-65-55:2013-08-29 12:02:37 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju juju-log.go:64 logstash-agent/1: Finished loading config variables
ip-10-151-65-55:2013-08-29 12:02:37 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Scan host for known log files and add logstash file inputs for them'
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Scan host for known log files and add logstash file inputs for them"]
ip-10-151-65-55:2013-08-29 12:02:38 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju juju-log.go:64 logstash-agent/1: Scan host for known log files and add logstash file inputs for them
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju context.go:221 worker/uniter: HOOK + '[' -d /var/log/apache2 ']'
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'remove apach2 input config'
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["remove apach2 input config"]
ip-10-151-65-55:2013-08-29 12:02:38 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju juju-log.go:64 logstash-agent/1: remove apach2 input config
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju context.go:221 worker/uniter: HOOK + '[' -f /opt/logstash-agent/conf.d/input_file_apache.conf ']'
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Check for Custom Log settings: '
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Check for Custom Log settings: "]
ip-10-151-65-55:2013-08-29 12:02:38 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju juju-log.go:64 logstash-agent/1: Check for Custom Log settings:
ip-10-151-65-55:2013-08-29 12:02:38 INFO juju context.go:221 worker/uniter: HOOK ++ config-get CustomLogFile
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["CustomLogFile"]
ip-10-151-65-55:2013-08-29 12:02:39 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + CustomLogFile='['\''/var/log/*.log'\'','\''/opt/storm/storm-0.8.1/logs/*.*'\'']'
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK ++ config-get CustomLogType
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["CustomLogType"]
ip-10-151-65-55:2013-08-29 12:02:39 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + CustomLogType=
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + '[' '!' -z '['\''/var/log/*.log'\'','\''/opt/storm/storm-0.8.1/logs/*.*'\'']' ']'
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + '[' '!' -z '' ']'
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Not Found! remove logstash input'
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Not Found! remove logstash input"]
ip-10-151-65-55:2013-08-29 12:02:39 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju juju-log.go:64 logstash-agent/1: Not Found! remove logstash input
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + '[' -f /opt/logstash-agent/conf.d/input_file_custom.conf ']'
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Check for Twitter input settings: '
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Check for Twitter input settings: "]
ip-10-151-65-55:2013-08-29 12:02:39 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju juju-log.go:64 logstash-agent/1: Check for Twitter input settings:
ip-10-151-65-55:2013-08-29 12:02:39 INFO juju context.go:221 worker/uniter: HOOK ++ config-get InputTwitterEnabled
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["InputTwitterEnabled"]
ip-10-151-65-55:2013-08-29 12:02:40 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK + InputTwitterEnabled=False
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK ++ config-get InputTwitterUsername
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["InputTwitterUsername"]
ip-10-151-65-55:2013-08-29 12:02:40 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK + InputTwitterUsername=
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK ++ config-get InputTwitterPassword
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["InputTwitterPassword"]
ip-10-151-65-55:2013-08-29 12:02:40 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK + InputTwitterPassword=
ip-10-151-65-55:2013-08-29 12:02:40 INFO juju context.go:221 worker/uniter: HOOK ++ config-get InputTwitterKeywords
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "config-get" ["InputTwitterKeywords"]
ip-10-151-65-55:2013-08-29 12:02:41 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + InputTwitterKeywords=
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + '[' False ']'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + '[' '!' -z '' ']'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Not Found! remove logstash input'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Not Found! remove logstash input"]
ip-10-151-65-55:2013-08-29 12:02:41 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju juju-log.go:64 logstash-agent/1: Not Found! remove logstash input
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + '[' -f /opt/logstash-agent/conf.d/input_twitter.conf ']'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + juju-log 'Restart logstash-agent'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["Restart logstash-agent"]
ip-10-151-65-55:2013-08-29 12:02:41 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "logstash-agent/1:config-changed:7678240702554379901"; dir "/var/lib/juju/agents/unit-logstash-agent-1/charm"
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju juju-log.go:64 logstash-agent/1: Restart logstash-agent
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK ++ initctl status logstash-agent
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK ++ grep running
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + '[' 'logstash-agent start/running, process 27455' ']'
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK + initctl restart logstash-agent
ip-10-151-65-55:2013-08-29 12:02:41 INFO juju context.go:221 worker/uniter: HOOK logstash-agent start/running, process 27593
ip-10-151-65-55:2013-08-29 12:02:42 INFO juju uniter.go:358 worker/uniter: ran "config-changed" hook
