Data and Observability Pipelines
-
Cyber Operations - Recon Blogs
Over the last few years, “observability pipelines” have become a fashionable abstraction. New vendors promise cost control, freedom from lock‑in, and routing flexibility. Those are all real problems, although in solving them, organisations are quietly re‑introducing the very fragmentation and operational risk they were trying to escape.
Many organisations are focusing on building centralisation. This doesn’t come without its difficulties. Most have concerns related to the collection of data, volume of data and the mechanisms related to its processing. Each of these is important and dependent on each other. Yet there is a core error in how they are reviewing this.
Centralisation Is Not About Collection. It’s About Authority
Observability and Security systems do not fail because of weak correlation engines. They fail because the data arriving at them is inconsistent, lossy, or semantically ambiguous.
Your ingestion method derives the Observability and Security contexts and parsing is critical to enforce normalisation:
- Deterministic pipelines
- Ordered filter execution
- Explicit error handling
- Persistent queues for back‑pressure and failure recovery
These characteristics are why Recon utilises Logstash as a core component of centralised logging and security pipelines, particularly in environments where auditability and replay matter.
Structure before Search: Why Observability and Security depend on Ingest Discipline
Observability and Security systems do not fail because of weak correlation engines. They fail because the data arriving at them is inconsistent, lossy, or semantically ambiguous.
Your ingestion method derives the Observability and Security contexts and parsing is critical to enforce normalisation:
- Deterministic pipelines
- Ordered filter execution
- Explicit error handling
- Persistent queues for back‑pressure and failure recovery
These characteristics are why Recon utilises Logstash as a core component of centralised logging and security pipelines, particularly in environments where auditability and replay matter.
Observability Is Converging - Logstash was built for that!
Modern observability is no longer split cleanly between logs, metrics, traces, and security events, users are increasingly expect converged telemetry, even if they present it through different lenses.
Logstash’s plugin‑driven model and pipeline isolation were early acknowledgements of this reality. It can ingest from heterogeneous sources, apply context once, and emit to multiple downstream consumers without duplicating logic or instrumentation.
This is a materially different design assumption than tools whose primary goal is routing optionality. Centralised pipeline management in Logstash, for example, explicitly assumes that configuration itself is a governed asset, not an edge‑level convenience.
Where “Observability Pipelines” Quietly Fall Short
I have heard of many various offerings for pipeline management from numerous vendors (most recently Cribl) over the years. It’s always the same message “Not all data deserves to be treated equally”. Its positioning around routing, sampling, and cost control resonates for good reason. The cost savings related to this can be easily felt by customers.
However, this same framing exposes several architectural gaps in their approach when applied to an enterprise‑grade observability model:
- Routing Is Not Governance – Flexibility of data in collection adds complexity in governance approaches. In true observability and compliance contexts, flexibility without a canonical processing layer becomes a liability.
- Edge‑Heavy Logic Increases Drift – Pushing transformation logic closer to sources improves latency, but it also increases configuration entropy.
- “Search in Place” Is Not Correlation – These vendors expand “Access to data” yet most don’t provide a uniformed event model. This increases investigation times exponentially and adds complexity and higher-level skills to hunt and search for data sets that match criteria.
These are not flaws so much as design trade‑offs. They matter when things go wrong, yet often these are not felt till that occurs.
Why Logstash is the logical choice.
Besides the fact that Logstash has been used as the backbone for many IT and OT aggregation points for many different Observability and Security environments (I have used it for Splunk, Elastic, New Relic, Data Dog, Dynatrace, Dragos, Nagios and more). There are 3 core reasons for this:
- Efficiency in Collection – Focused on ensuring data is structured rather than optionally applied. Enterprise-grade centralised observability focuses on clear alerting in a structured way.
- Free to use – Logstash has been free since its inception in 2012. You can horizontally scale and adapt based on your workloads and the only thing you need to consider is the underlying infrastructure to run it.
- Ingest Layer Durability – With the use of persistent queues and dead-letter, events and information survive even if there is a downstream failure. No need to rehydrate.
When paired with Elastic, Logstash becomes even more powerful as the pipelines can be managed centrally so that you can make modifications to data collection/filtering and classifications all via the GUI.
Pairing Logstash and Elastic
Logstash and Elastic make centralised observability and security easy to buy, easy to deploy, and easy to justify because they solve multiple problems with one platform.
You don’t need to assemble a complex toolchain to collect, process, store, and secure data.
- Logstash centrally normalises and enriches telemetry once.
- Elasticsearch makes it instantly searchable at scale
- Kibana turns it into dashboards, alerts, and detections.
The combination of Elastic and Logstash removes friction where it matters most: cost, complexity, and risk. The same data powers observability and security use cases without duplication, re‑ingest, or separate platforms, which directly lowers total cost of ownership.
Through Recon’s work around pipelines and pipeline management, we leverage Logstash to build efficiency and reduce the processing of data at the Elastic cluster level. As a result, we have been able to ingest 2-3 x the data without additional Elastic licensing costs for our clients.
Conclusion
Centralised observability and security don’t fail because teams can’t collect or route data. They fail when no one owns what that data means. Treating ingest as neutral or purely flexible pushes complexity downstream, where it becomes harder to govern, harder to correlate, and more expensive to fix when things go wrong.
Logstash endures because it was built to provide that ownership. It delivers a durable, deterministic, and governed ingest layer that organisations can trust as observability and security converge. When paired with Elastic, data is structured once, reused everywhere, and trusted across operations, security, and compliance. Centralisation isn’t about moving data faster or cheaper. It’s about authority. Logstash and Elastic make that authority simple, scalable, and reliable.
Latest News and Insights
PKI and Certificate Management the risk most miss.
An expired certificate can take down critical business services faster than many cyber attacks. With certificate lifecycles shrinking, PKI...
Read MoreData and Observability Pipelines
Observability pipelines have become a fashionable abstraction although by going down this route organisations are quietly re‑introducing the very...
Read More
Recon Partners with CrowdStrike to Elevate Cybersecurity for Australian Businesses
Recon has partnered with CrowdStrike to bring AI-powered cybersecurity to Australian businesses, delivering advanced threat detection and response through...
Read More
Google and M365 Security
Google and Microsoft offer great solutions for collaboration, yet the intricacies of these platforms need to be understood to...
Read MoreLooking to transform your cloud and cyber practices?
Recon transforms cloud and cyber from cost and risk into drivers of growth. Our focus is enabling scale, protecting value, and supporting confident business expansion.