Onboarding Azure Data into Splunk

As organisations increasingly adopt Azure for their cloud computing needs, integrating Azure data into Splunk becomes critical for effective monitoring, security, and compliance. This blog explores the three core categories of Azure services; Infrastructure as a Service (IaaS), Software as a Service (SaaS), and Platform as a Service (PaaS) – detailing their significance, the type of logs to onboard, and the engineering approaches for ingesting data into Splunk.

Infrastructure as a Service (IaaS)

IaaS in Azure encompasses virtualised infrastructure such as virtual machines (VMs), virtual networks, Azure Storage, and load balancers. Onboarding logs from IaaS services into Splunk is vital for monitoring infrastructure health, security, and performance.

Key Log Types

• Activity Logs: Track infrastructure-level changes, e.g., VM creation, scaling events.
• Azure Monitor Insights: Provide performance data and disk metrics.
• Network Security Group (NSG) Flow Logs: Analyse network traffic for suspicious activity.
• Diagnostic Logs: Capture service-specific data for troubleshooting.

Use Cases

• Infrastructure Health Monitoring: Ensure uptime and optimal performance of VMs and other resources.
• Security Monitoring: Analyse NSG flow logs for potential threats, e.g., anomalous network activity.
• Monitoring Changes: You can monitor changes via activity logs e.g. infrastructure being deleted.

Why It Matters

IaaS logs in Splunk empower teams to maintain robust infrastructure performance and enhance security through detailed insights.

Software as a Service (SaaS)

Azure SaaS offerings, including Office 365, Dynamics 365, and Azure AD, provide critical business and productivity tools. Onboarding their logs helps organisations address identity, access, and compliance needs.

Key Log Types

• 365 Security and Compliance Logs: Monitor user activity and detect anomalies.
• Azure AD Sign-In and Audit Logs: Track logins and configuration changes.
• Exchange logs: To do email use cases and look for data exfiltration etc.
• SharePoint Logs: Monitor data exfiltration or sensitive documents being viewed/modified.

Use Cases

• Identity and Access Management (IAM): Monitor for unusual login patterns, privilege escalation, and policy violations.
• Compliance Auditing: Ensure adherence to regulations by reviewing user activity logs.

Why It Matters

Integrating SaaS logs into Splunk enables organisations to enhance security, streamline compliance processes, and conduct detailed investigations into user behaviour.

Platform as a Service (PaaS)

PaaS includes managed services like Azure SQL Database, Azure App Service, and Azure Kubernetes Service (AKS). Although less frequently onboarded, PaaS logs provide unique insights into application and resource performance.

Key Log Types

• Application Insights Telemetry: Monitor application performance and availability.
• Diagnostic Logs: Collect request and execution logs for troubleshooting.

Use Cases

• Application Performance Monitoring: Identify bottlenecks in app and database performance.
• Resource Usage Optimisation: Track metrics to manage resource consumption effectively.
• Security Monitoring: Detect SQL injection attempts or other suspicious database interactions.

Why It Matters

Onboarding PaaS logs enables deeper visibility into application-level issues and back-end service performance, making it a powerful tool for both production monitoring and security enhancement.

Engineering Considerations for Ingesting Azure Data into Splunk

Bringing Azure data into Splunk can be achieved using either the pull or push method, each with its own benefits and trade-offs.

Pull Method

• Overview: Uses pre-configured Splunk technology add-ons (TAs) to fetch data via API connections.
• Advantages:
  • Low configuration overhead.
  • Easy integration, especially for Splunk Cloud users.
  • Ideal for smaller deployments.
• Limitations:
  • Runs on a schedule, introducing delays.
  • Limited scalability for high-volume environments.
  • Requires more management/maintenance.

Push Method

• Overview: Leverages the HTTP Event Collector (HEC) to push logs directly from Azure to Splunk.
• Advantages:
  • Near real-time log ingestion.
  • Highly scalable for large environments.
  • Lower management/maintenance overhead compared to pull method – despite higher configuration overhead.
• Limitations:
  • Requires significant configuration effort.
  • Necessitates DevOps expertise in programming and Azure services.

Recommendation

For small-scale setups, the pull method offers simplicity and ease of use. In contrast, the push method is more suitable for large-scale environments requiring real-time logging and advanced customisation.