Mastering Logstash with e2e testing

By Meabed on 2018-10-01 #docker #docker-machine #logstash #testing #tdd #devops #elk #e2e #logstash-testing

Logstash is very flexible and rich powerful engine used in every organization, but to unlock its powers you have to MASTER IT.

Why UI?

Testing Logstash real configuration input, output, codecs and filters is a extremely tedious process results in logstash configurations that are not tested properly and it break very often.

To master it, you need a predictable testing configuration.

GO + Socket.IO + NG + Logstash FTW!

Logstash_Testing

In this post i will demonstrate how to achieve e2e testing for Logstash server configuration.

Logstash components as it explained in the docs:

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.) https://www.elastic.co/products/logstash

Logstash is used in almost every tech team with main users as devops or sysadmins, but hey do you know that we use Logstash in so many other ways? like: application log shipping, different server logs, ingest and transform data then output it to any destination such as "database / s3 / BI or data warehouse, etc..."

But configuring Logstash is usually a hassle and takes a lot of effort and based on trial an error!

Mastering Logstash Testing

In this examples we will use

  • docker, docker-compose to run Logstash
  • HTTP Input to send our test data to Logstash through HTTP "Receives events over HTTP or HTTPS"
  • Some filters to demonstrate the filter usage like:
  • Stdout to output the transformed data so we are able to determine the testing result "Visually"

You can check out the example in GITHUB REPO and below some screens demonstrate how the project works

Architecture

Logstash architecture in production

Logstash_Testing

Logstash architecture in this example

Logstash_Testing

Screenshots

Logstash_Testing

Logstash_Testing

Check it out HERE

...