Remember: different types of filter plugins exist for different processing needs.įor example, there are plugins for parsing and processing XML, JSON, unstructured, and CSV data, API responses, Geocoding IP addresses, or relational data. In this stage, necessary data elements are extracted from the input stream. Once data is ingested, one or more filter plugins take care of the processing part in the filter stage. Note: There’s a multitude of input plugins available for Logstash such as various log files, relational databases, NoSQL databases, Kafka queues, HTTP endpoints, S3 files, CloudWatch Logs, log4j events or Twitter feed. Logstash itself doesn’t access the source system and collect the data, it uses input plugins to ingest the data from various sources. In the input stage, data is ingested into Logstash from a source. In each stage, there are plugins that perform some action on the data. Again, there are prebuilt output interfaces that make this task simple.ĭata flows through a Logstash pipeline in three stages: the input stage, the filter stage, and the output stage. Logstash can easily parse and filter out the data from these log events using one or more filtering plugins that come with it.įinally, it can send the filtered output to one or more destinations. To Logstash, all these data are “logs” containing “events”. These data can be structured, semi-structured, or unstructured, and can have many different schemas. Unlike ETL jobs though, Logstash is a generic engine which means it can accept data from many different sources out-of-box. This is what an ETL (Extraction, Transformation, and Loading) job will do. Usually, custom-developed data pipelines extract specific types of data from specific sources, perform some predefined actions on the data, and then save the result in a specific location.
But there’s more to it than typical pipelines that data engineers develop. This makes Logstash essentially a data pipeline. These applications collect logs from different sources (software, hardware, electronic devices, API calls, etc.), process the collected data, and forwards it to a different application for further processing or storing. Logstash is typically used as the “processing” engine for any log management solution (or systems that deal with changing data streams). In most cases, the downstream system is Elasticsearch, although it doesn’t always have to be that, as we will learn later. The classic definition of Logstash says it’s an open-source, server-side data processing pipeline that can simultaneously ingest data from a wide variety of sources, then parse, filter, transform and enrich the data, and finally forward it to a downstream system.
0 kommentar(er)
