Logstash and Log4j

November 11, 2014

I wanted to log from a log4j process through to Logstash, and have the logging stored in Elastic search. This can be done using the code at https://github.com/logstash/log4j-jsonevent-layout

To make things easy for my test, I put the source code for net.logstash.log4j.JSONEventLayoutV1 and net.logstash.log4j.data.HostData into my source tree.

I then added json-smart-1.1.1.jar to the classpath (from https://code.google.com/p/json-smart/downloads/detail?name=json-smart-1.1.1.jar&can=2&q=)

I then created the following log4j appender in log4j.xml then used it in code:

<appender name="LogStash"
    <param name="File" value="logs/panCoreSaas.log" />
    <param name="Append" value="true" />
    <layout class="net.logstash.log4j.JSONEventLayoutV1"/>

Using this appender produces logs in JSON format:

{  "thread_name":"http-8080-3@#zUaiBHqDI66O25BntXrghixzzEljOHlHGPB0dNmz4RoX3gNWyB",
    "message":"Calling FindEvents",
    "mdc": {},

Create a log4j_file.conf with the following contents:

input {
  file {
    path => "/apps/logstash/logstash-1.4.2/panCoreSaas.log"
    start_position => beginning

output {
  elasticsearch {
    host => localhost
  stdout { codec => rubydebug }

and then start logstash to process the file

$ bin/logstash -f log4j_json.conf

Tags: logstash log4j