Case Study: Licence Growth and Role Management in Splunk Environment

Overview

This organisation approached us with a pressing issue regarding the size and growth rate of their Splunk licence and its predictability. As a data-driven organisation, they were ingesting large amounts of data but were not managing or storing it efficiently. The lack of a standardised approach to managing their indexes and roles in Splunk was driving their licence growth at an unsustainable pace, leading to spiraling costs. Despite carrying out remediation activities to control both ingest and retention, a poor RBAC policy was leading to unpredictability in ingest and storage volumes. Our goal was to help them reduce their licence size, optimise their data management processes and bring predictability to their licence usage.

The Challenge

The company’s primary concern was the rapid and erratic growth of their Splunk licence. Their users were ingesting data without proper oversight from system admins, which led to inconsistent licence increases. This uncontrolled ingestion not only inflated their costs but also complicated their Splunk environment.

Moreover, we uncovered a broader issue during our investigation: their role-based access control (RBAC) system within Splunk was spread across many different roles, resulting in confusion and inefficiencies. Users were granted access to ingest data without following a proper approval process and there was a lack of accountability across the departments using the platform. This uncontrolled access was costing the company more money and complicating the management of their Splunk environment.

Why They Chose Us

The company sought our help due to our deep knowledge of Splunk’s data feeds along with our expertise in networking and licence management. We were able to jump straight into the problem-solving process with minimal guidance from the client. Our ability to propose immediate solutions to work around the growing licence costs, as well as proposed a more robust approach over the longer term made us an ideal partner for this engagement.

Approach and Solution

To tackle the licence growth, we began by creating a model to analyse their current licence usage. Our model identified which indexes and roles were contributing most to the data being ingested into Splunk. This allowed the client to identify key areas where they needed to reduce data ingestion. One critical aspect we focused on was the retention rates of their data, which had a direct impact on the size of their licence. Many individual users in the business weren’t aware of how their retention settings affected data storage so we presented the findings in a way that allowed them to interact with the data and simulate different retention settings.

Given that this customer didn’t have a separate testing environment, we modeled these changes outside of their live environment to mitigate any risks of losing important data. This model helped them understand the effects of reducing retention times without affecting their live system.

Additionally, we created a technical note outlining how they could offload the data causing excessive storage costs to AWS. By doing this, they could store less critical data externally and bring it back into Splunk only when needed which reduced costs significantly.

We also discovered inefficiencies in their RBAC system. As a result, we worked on streamlining the roles in their Splunk environment by implementing a more structured role-based access control system. This involved reviewing the roles, speaking with stakeholders and understanding how different users interacted with the platform. Ultimately, we consolidated roles and implemented a more efficient access control system to prevent unchecked data ingestion and improve system management.

Outcome

As a result of the engagement our customer now has a better understanding of their licence growth and the underlying causes. They are now spending significantly less on their Splunk licence and the systems team is empowered to manage data ingestion more effectively. Additionally, with a restructured RBAC system, the company has greater control over who is using Splunk, what data they are ingesting and how it’s being stored.

Business Impact

From a business perspective, the company can now avoid unnecessary licence increases and has a clearer understanding of future licence requirements. This has led to a more cost-efficient, predictable operation with the systems team better equipped to communicate with other departments about their Splunk usage. Furthermore, this process has prompted them to take a more critical look at how Splunk is used across the organisation, ensuring that data ingestion is intentional and managed properly.

Recommendations for Other Splunk Users

For other Splunk users facing similar challenges with unpredictable ingest and storage retention, we recommend the following:

• Understand Your User Base: Ensure there is a clear understanding of who is using Splunk and why. Overloading the platform with users without proper oversight can lead to significant inefficiencies.
• Manage Data Retention: Regularly review and adjust your retention policies. Data retention has a significant impact on your licence size and costs.
• Educate Users and Admins: Provide training for both users and system admins on how to use Splunk efficiently. This can prevent costly mistakes, such as excessive storage costs due to improper use of safe searches and lookups.
• Centralise Role Management: Ensure that access control and data ingestion responsibilities are well defined and that roles are consolidated and structured to avoid confusion and inefficiencies.