I am now doing some projects that need a monitoring application to monitor the webservice. After having some chit and chat, we decide to use ELK (Elasticsearch, Logstash, and Kibana). If you want to know what ELK is, just search on Google and there will be so many articles related to it.
If you have already read some articles about ELK, you will know that ELK is the application to monitor and analyze all types of log.
- Elasticsearch: indexing the data.
- Logstash: log processing / parsing.
- Kibana: visualize the data.
But after trying to configure and run ELK, I found out that Logstash is heavy to be run on the server with small specification. Because of this reason, I am trying to find some Logstash alternatives, and finally I found Gogstash, Logstash like, written in Golang.
While reading the documentation, I found out that there are some differences between Gogstash and Logstash when using the filter (I am using grok filter in Logstash). I tried to apply same pattern in Gogstash but it didn’t work. After all these things, I decide to use another filter. I am using gonx filter.
Although grok pattern and gonx pattern is different, it is not so difficult to create the configuration for gonx filter. And after some modification, Gogstash run smoothly. For your information, I am using flask for building the webservice, and this is an example line of the application log.
192.168.100.57 - - [05/Dec/2017 16:27:27] "GET / HTTP/1.1" 200 -
There are two types of Gogstash configuration, json and yml format. Here is my yml configuration.
input:
- type: file
path: '/home/linggar/webapp/nohup.out'
filter:
- type: gonx
format: '$clientip - - [$date $time] "$full_request" $status -'
source: message
- type: gonx
format: '$verb $request HTTP/$httpversion'
source: full_request
output:
- type: elastic
url: 'http://127.0.0.1:9200'
index: gogstash_log
document_type: testtype
And that’s it. Although Gogstash is not as powerful as Logstash, it is very light and one of so many Logstash alternatives which you could try.