Ingest Actions

Overview

As part of our operate support package, we conducted a data discovery piece for a customer and provided recommendations on ingest actions as they were in a predicament. They were periodically exceeding their licence limit, but were yet to onboard some key data sources which they felt could be valuable to the business and improving their security posture. In addition to ingest challenges, they were also approaching the limits of their searchable storage. Opposing pressures meant that their budget could not be increased to accommodate more licence spend within the year. To address this, we conducted a data discovery exercise (our approach to this is detailed in a separate blog) to see how we could help.

Our Approach/Solution

Our data discovery approach gave a detailed breakdown of the data the customer had in their Splunk environment along with who, when and what it was being used for. Critically the work helped the customer understand the drivers behind their periodic spikes in ingest. In this case, the customers environment primarily consisted of Windows Event Logs from Windows servers and endpoints, along with a considerable volume of antivirus logs, system monitoring data, and some firewall logs. Given this, we focused on reducing the volume of winevents using Ingest Actions.

To achieve this, we:

• Reviewed ingested data by index and time period, analysing patterns and forecasting future storage increases.
• Mapped the data used in their Enterprise Security (ES) implementation to ensure their main use case for Splunk remained supported.
  This process allowed us to determine which data sources were essential for their use cases and which could be optimised or filtered out.

What we learned

We identified the specific event codes used in their searches. Out of 477 event codes in the data, only 110 were being utilised. This revealed that up to 75% of the data was potentially unnecessary for their ES use cases. We provided the customer with a comprehensive report and delivered a presentation outlining our findings and recommendations. This included identifying winevent codes that were not being used and suggesting optimisations. The customer implemented our recommendations using ingest actions to filter out unused logs that were not required for their ES use cases.

Using the output of our discovery process we also worked with the client to define a fit for purpose retention strategy, enabling the customer to optimise the value from their storage entitlement.

Result of the Implementation

After implementing Ingest Actions in a considered way, we successfully controlled the customers ingest patterns and freed up additional licence to being bringing additional data sources into Splunk.

As a result of onboarding this new data, the customer was able to increase their Enterprise Security usage, increasing their coverage with new use cases, and enhancing their overall security posture. This approach meant that ultimately, the customer was able to demonstrate a far more compelling picture of the value Splunk was brining to the business.

Ongoing Management – enabling the customer to stay on top of their data

Our customer had subscribed to our Operate service, giving us the opportunity to help them with the issues they faced. A great thing about ingest actions is each data transformation is configured as a rule, multiple rules can be applied to a data set, and then saved as a ruleset of combined rules.  In distributed environments, this ruleset can be deployed from the cluster manager to the indexer clusters, or from the Deployment Server to the Heavy forwarders, or simply on the Splunk Cloud search head and deployed to the indexers – one configuration taking effect across the distribution. Using this approach, we were easily able to take on the management of these rulesets as part of our Operate service, helping the customer keep on top of what data they did and did not need and ensuring the rulesets were always up to date, in line with their requirements and licence.