Optimize the bill, not the record.

The economics that created the optimization market are simple. SIEM platforms price by ingestion, the meter runs on every gigabyte, and telemetry volume grows faster than any security budget. We walk the numbers in why SIEM ingestion cost keeps rising and in our comparison of SIEM pricing models: past a certain scale, the invoice becomes the design constraint for the entire security architecture.

Into that pain stepped a category of telemetry pipeline products whose job is to shrink the stream before the meter sees it. The functionality is consistent across vendors: filter events that look uninteresting, sample noisy sources, drop verbose fields, aggregate detail into summaries, deduplicate, and route the remainder to cheaper destinations. Teams adopt them for a rational reason, the SIEM bill falls immediately and visibly, often by half or more.

It is worth being precise about what these tools do well. As routing and transformation infrastructure, they are genuinely useful: getting the right stream to the right destination in the right shape is a real problem. The failure is in the central promise, that the way to afford your telemetry is to have less of it. That promise deserves scrutiny, because the thing being reduced is not cost. It is the record.

The storage problem is real: full-fidelity telemetry at enterprise scale is expensive to keep in platforms that were never engineered for it. Filtering does not solve that problem. It deflects it, by making the data small enough to fit inside a broken cost model, and it pays for the deflection with the data itself.

Every filter rule is a prediction about the future. It says: no question we ever ask will need this event, no detection will ever key on this field, no investigation will ever pivot through this identity at this hour. Those predictions are made at capture time, by whoever tuned the pipeline, under budget pressure, and they are permanent. An event that was filtered is not archived, not tiered, not deferred. It never existed. There is no restore job for a record that was never written, which is what separates filtering from every other cost lever we cover in five ways to reduce SIEM costs without sacrificing visibility.

This is why we say filtering is deflection. The invoice improved because the memory got worse, and the organization has quietly agreed that its telemetry, the thing we argue in enterprise telemetry is organizational memory is a compounding asset, is actually a liability to be minimized. A system of record that has been optimized this way is no longer a record. It is a highlight reel.

The gaps are not a theoretical future cost. They are collected continuously, by every consumer of the thinned record. Detection pays first: a rule that needs the dropped field cannot disambiguate the signal, so it either misses or it floods, the fidelity argument we develop in security detection on full history. The signal that separates a hostile process from a benign one often lives in exactly the verbose fields, command lines, call stacks, parent chains, that optimization pipelines trim first, as our work on call stack detection in Sysmon shows concretely.

Investigations pay next. A causal reconstruction is only as complete as the record it walks, and a sampled stream breaks the chain at whichever hop the sampler discarded, the failure mode we describe in investigations and forensics. And compliance pays at audit, when the honest answer to "is the record complete" becomes a description of the filter configuration, turning the audit into the negotiation we warn about in compliance and audit history. Sampling was a survivable compromise when the only consumers were dashboards. Every consumer since has made it more expensive.

The heaviest cost lands on the newest consumer. Every security and observability stack is currently being wired for AI: agents that triage, investigate, decide, and act by reasoning over telemetry. Those systems inherit whatever record the pipeline left behind, and unlike a human analyst, they cannot compensate for gaps with intuition about what is probably missing. The case we make in agentic AI security needs memory, not just models applies with full force here: the model is interchangeable, the memory is not.

Incomplete telemetry does not degrade an AI system gracefully. Confidence drops drastically, because confidence is a function of evidence coverage. An agent asked whether an identity has behaved this way before can only answer from the events that survived the filter, so its conclusion carries an invisible asterisk: based on the fraction of reality we kept. Worse, a filtered record breaks the provenance that makes automated decisions defensible at all. A conclusion that cannot cite the record, the standard we set in AI-driven decisions and in the explainability gap, cannot be replayed, audited, or trusted with an action.

This is the strategic error in the optimization wave. Complete data cannot be filtered and remain complete, and machine consumers are the first audience for whom completeness is not negotiable. The organizations spending this budget cycle tuning filters are preparing their telemetry for the previous era's consumers, precisely as the next era's consumers arrive, the argument of the agentic data plane and AI agent memory for the enterprise.

The reason filtering feels necessary is that the underlying storage economics feel immovable. They are not. Datafabric attacks the storage problem directly: telemetry is captured at full fidelity through the capture pipeline, compressed into an open, ultra-efficient format, and retained hot for years on object storage inside your own cloud, with economics that scale with time rather than volume and no ingestion meter anywhere in the path.

Under that model, the question every filter rule tries to answer, is this event worth keeping, stops being worth asking. Keeping everything is not an act of discipline or of budget heroics, it is the default, the same way version control keeps every commit. The principle is the one we define in full-fidelity log retention: the record is complete by construction, so no future consumer, human or machine, inherits a decision made under last year's budget pressure. That is what it means to solve the problem rather than deflect it. The cost curve is fixed at the substrate, not negotiated event by event at the edge.

None of this means surrendering the outcome that made optimization attractive. The legitimate goal, spending less on the SIEM, survives intact. What changes is the mechanism. Instead of thinning the stream before it reaches storage, the full record lands in Datafabric as the system of record, and the SIEM becomes what it always should have been: one downstream consumer among several, reading only the slice it needs for the workflows that still live there. The architectural inversion is the subject of Bloo as the system of record and our analysis of SIEM versus security data lake architectures.

The difference from a filtering pipeline is exact: reduction happens downstream of the record instead of upstream of it. The SIEM ingests less, so the bill falls, but every event the reduction skipped still exists, hot and searchable across years, available to detection, investigation, audit, and every agent that comes later. Reduction becomes a routing decision you can revise, not a deletion you must live with.

Teams that take this path usually start where the pain is, the approach behind our enterprise logging solution: land the full stream in the fabric first, point the expensive consumers at reduced reads, and let the legacy platform shrink into the use case it is actually good at. Optimization was never the wrong instinct. Optimizing the record instead of the architecture was the wrong target.