I wanted to have a play with LogStash to see if it would help with analysing log files. I first followed the instructions at http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash, which covers settings up logstash, running elastic search and logging events to it.

• Elastic search is a data store
• Logstash is the process by which logging information gets into that data store
• Kibana is a tool used to query elastic search and extract log information

Kibana

To get Kibana working, download it and copy it to a webserver. Edit config.js to point to your elasticsearch server:

elasticsearch: "http://linux07:9200"

You can then navigate to the directory on the webserver from a browser, and Kibana will run in the browser.

Reading Apache Logs

To read apache logs into the setup, create the following as apache_log.conf:

input {
  file {
    path => "/var/log/httpd/access_log"
    start_position => beginning
  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { "type" => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
   elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}

and then run it using

$ bin/logstash -f apache_log.conf

Using Kibana

Start Kibana in a browser by navigating to it, then choose the Logstash Dashboard option in the bottom right.

The screen shown will look something like this:

You can drill into the graph by clicking and dragging to draw a box.

If you scroll down the page you can see details of the logs that are shown in the graph above, and you can control which fields are shown using the list of checkboxes on the left hand side.

Click on the name of the checkbox to show a summary of counts

Searching

To search, use the search box at the top of the page and the bottom will show a graph and list of messages that match the search criteria.