The security team’s goal is to develop a unified analytics platform that enables timely and comprehensive incident investigation at the F100 financial services organization. The team detects anomalous activity by searching for indicators of compromise over real-time and historic data. Logs from networking and security devices are streamed to their SIEM (RSA Security Analytics platform) which produces AVRO files. The team originally deployed a Hadoop-based solution to search and analyze the AVRO data but they faced significant limitations, including a 24-hour lag time before they could query the data and many of their queries were taking hours to run due to the massive data volumes. X15 Enterprise™ was implemented and provided the following benefits:
- Enabled real-time search and analysis
- Improved query precision
- Expedited anomaly detection
- Provided user behavior analysis (UBA)
- Improved query performance
- Enriched data with additional context for incident investigations
- Increased operational efficiency
- Reduced total cost of ownership
“The challenge that we have with our existing tools is that it is easy to search for a single IP address, MD5 hash, or user ID. But what is hard to do is search for a list of them. X15 is the only product in the market that enables us to rapidly search petabytes of logs for indicators of compromise within seconds.”
Director Security Operations,
F100 Financial Services Organization
This F100 financial services organization employs over 50,000 globally which makes security extremely challenging despite having a security team of several hundred professionals. Logs from over 70 different device types which include servers, workstations and security devices are streamed to their RSA Security Analytics platform to produce AVRO files. The security team’s goal is to detect anomalous activity by searching for known bad IP addresses or domains, MD5 hashes of files, or risky users over several months.
Hadoop-based solution was complex, slow, and introduced significant latency
The security team faced some challenges caused by data blind spots. Given the large variance in log formats across systems, the RSA platform was not able to capture all the fields. This resulted in critical data going unidentified in the parsed AVRO file.
To mitigate the blind spots, both the parsed and raw AVRO files were streamed to a HIVE cluster for analysis. This created a 24-hour lag time due to the overhead associated with transforming the data and loading it into an analytics schema in Hadoop. The approach was less than ideal because it inhibited real-time analysis. To make matters worse, the queries performed very poorly due to the size of the data set.
Alternatives such as Splunk and ELK were not a good fit
The team looked at a variety of other solutions but none of them were a good fit. While security analytics solutions produce insights into threats based on intelligent algorithms, they operate as black boxes and provided little-to-no ability for the team to perform its own analysis. The ELK stack was also evaluated but could not natively ingest AVRO files. Splunk was also under consideration but the complexity of the queries and cost were deterrents. Other open source solutions were also evaluated but did not meet the criteria.
The X15 Solution
X15 Enterprise seamlessly replaced the Hadoop-based solution
The security team ultimately selected the X15 platform. In the new architecture, X15 ingests the AVRO files from the SIEM and the streaming raw logs from a Kafka bus at a rate of 8TB per day. This completely replaced 40 HIVE nodes creating significant cost savings. The security team uses the X15 application to search petabytes of logs for bad actors in seconds.
Enabled real-time search and analysis
The X15 platform discovers data structure upon ingestion and dynamically adjusts it over time to eliminate the need to specify rigid schemas upfront. This is critical since the logs from different device types can vary greatly. It also eliminates the 24-hour lag time associated with the overhead of transforming the data and having to load it into an analytics schema in Hadoop.
The security team can now search and analyze the data as soon as streaming data is ingested. As a bonus, any changes to the fields in the SIEM are transparently reflected in X15 without requiring any manual updates by the security team.
Improved query precision
The X15 platform stores both the parsed data from the AVRO file as well as the raw logs, boosting the accuracy of search queries. This approach ensures that any missing data in the AVRO file is accounted for in the raw logs.
Expedited anomaly detection
The X15 platform seamlessly integrates full-text search with structured data analysis to expedite anomaly detection. For example, a user can create a reference table from a file containing a list of entities such as known bad IP addresses; execute a single search query using the table; and locate all events involving these IPs. In traditional search and analytics tools, each entity would require its own query, making the search very cumbersome, especially as threat intelligence lists update frequently. With X15, the security team can search petabytes of logs for bad actors in seconds.
Provided user behavior analysis (UBA)
The X15 platform supports user and entity behavior analysis through a set of powerful windowing and sequence analysis functions. Once a bad actor is identified, the security team is quickly able to monitor its behavior and gain context about the breach including potential damage to their environment.
Improved query performance
The X15 platform performs both full-text as well as value-based indexing of data upon data ingestion. This enabled the team to perform search as well as more complex SQL query operations on billions of records in seconds. In contrast, SQL on Hadoop solutions perform full-table scans for similar queries, which makes them several orders of magnitude slower.
Enriched data with additional context for incident investigations
The X15 platform also collects and correlates data from non-streaming enterprise data stores. It treats the data stores as external tables and performs lookups on user IDs, MD5 hashes, and much more.
Increased operational efficiency
The dashboards and workflows in the X15 platform improve operational efficiency. Custom dashboards are used to continuously monitor for bad actors. The security team is alerted and scripts are triggered when anomalies are detected.
Reduced total cost of ownership
The X15 platform is deployed as a distributed cluster where every node in the cluster indexes, stores, and queries data to enable fault tolerance and true linear scalability as more nodes are added. A small X15 footprint replaced 40 HIVE nodes creating significant cost savings for the organization. The X15 platform’s ability to perform both search and analysis on the data without having to replicate it into another analytics platform enabled the team to cut their storage costs in half. Moreover, it optimizes redundant data by applying intelligent compression resulting in additional storage savings. Finally, X15’s affordable node-based licensing model ensured that the revamped analytics architecture could handle the growth in their daily data volumes without breaking the bank.