What is the best way to write log message of my java app into the elk stack?

Question

I would like to save the log data from my Java application into an ELK stack. I can configure the format of the log data freely. Unfortunately I can only use log4j 1.x.

What is the easiest way to store log data structured in ELK Stack?

Answer 1

If you asking for an easy way to ingest your Java app logs into Elasticsearch then install & run filebeat to read your app logs and ingest them into ES. You can use filebeat ingest node to be able to parse your logs and store them into individual fields in Elasticsearch to help with visualizing etc. Alternatively, you can also install logstash to do parsing instead of filebeat ingest pipelines. Some links for reading to get you started:

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html

https://www.elastic.co/guide/en/logstash/current/configuration.html

Answer 2

You can write your application level logs via RollingFileAppenders using Filebeat Add the below to your logback.xml to write app logs to a directory:

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/etc/logging/sample.log</file>
        <immediateFlush>false</immediateFlush>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>
                /etc/logging/hsvt_akka.%d{yyyy-MM-dd}.%i.log.zip
            </fileNamePattern>
            <maxFileSize>5KB</maxFileSize>
            <maxHistory>20</maxHistory>
            <totalSizeCap>1MB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>[%date{ISO8601}] [%level] [%logger] [%marker] [%thread] - %msg MDC: {%mdc}%n</pattern>
        </encoder>
    </appender>

Now configure filebeat to read these logs written to directory /etc/logging, running as a sidecar to your java application in the cluster, for eg k8s

Create configMap for filebeat:

filebeat.yml: |
    filebeat:
      inputs:
      - type: filestream
        enabled: true
        paths:
          - /etc/logging/*.log
          - /etc/logging/*.log.*
        prospector:
          scanner:
            check_interval: 1s

      config:
        inputs:
          path: /etc/filebeat.yml
          reload:
            enabled: true
            period: 10s
    output:
      elasticsearch:
        hosts: []
        protocol: 
        username: 
        password: 

Now add this as a sidecar container to your original java app.

- name: filebeat-sidecar
          image: elastic/filebeat:7.16.3
          command: ["filebeat"]
          args: ["-c","/etc/filebeat.yml","-e", "-strict.perms=false"]
          resources:
            limits:
              memory: "80Mi"
            requests:
              memory: "60Mi"
          securityContext:
            runAsUser: 2000
          volumeMounts:
            - name: filebeat-app-logs
              mountPath: /etc/logging
            - name: filebeat-config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
      volumes:
        - name: filebeat-app-logs
        - name: filebeat-config
          configMap:
            defaultMode: 0600
            name: filebeat-configmap
            items:
              - key: filebeat.yml
                path: filebeat.yml

You need to configure the java app and filebeat to read the same volume where the logs are written to