Threat Hunting Methodologies: Techniques and Best Practices

Threat hunting encompasses a range of techniques and approaches aimed at discovering anomalies, threats, and risks associated with attacker activities. In the early days, log review by diligent system administrators was how these anomalies were detected, usually after the fact. This evolved into more structured methodologies created by security experts that attempted to identify these activities in real time. In present day security operations, threat hunting initiatives have become a standard part of mature security programs, but few organizations have managed to establish the expertise and methodology to conduct these types of hunts with internal resources.

In this series, we will take a look at the components that make up well-known threat hunting methodologies, the evolution that reflects the growing need to proactively seek out and mitigate security threats rather than solely relying on reactive, manual measures, and some new adaptive approaches to conducting automated, wide ranging hunt capabilities.

Threat hunting, after all, involves implementing innovative methods of continuous monitoring and analysis of real-world activities to uncover hidden threats, making it an essential aspect of modern cybersecurity that should leverage every aspect of process, technology, and people available to defenders.

Traditional Methodologies

The approach advocated by threat hunting pioneers in the last decade emphasizes proactive cybersecurity practices. It involves the systematic and continuous search for hidden threats and anomalies within an organization’s environment, aiming to detect and mitigate potential breaches before they can cause damage.

In most cybersecurity practices, a robust approach involves utilizing a range of advanced tools. These tools encompass intrusion detection systems (IDS), Security Information and Event Management (SIEM) platforms, Endpoint Protection, Detection, & Response platforms (EPP/EDR), as well as threat intelligence feeds and security service providers. However, the effective application of these tools and services requires the expertise of seasoned cybersecurity professionals and highly tuned, effective tooling.

Experience allows security teams to leverage their knowledge, intuition, and subject matter expertise to interpret data and discern nuanced threats within logs, packets, flows, and trace activities. Some key elements found in traditional methodologies which deliver acceptable results but need to be constantly revisited and enhanced include:

• Structured Methodology: important for identifying anomalies and potential threats in network traffic, endpoint telemetry, and system logs.
• Continuous Monitoring: Real-time monitoring of network and system activities to detect suspicious or unusual behavior (security monitoring).
• Hypothesis-Driven Analysis: Develop hypotheses about potential threats based on known attack patterns, trends, or indicators, then investigate these hypotheses to confirm or refute their suspicions (threat modeling).
• Data Analysis: Scrutinizing large volumes of data, including network traffic logs, system logs, security controls’ logs, and other relevant data sources, to uncover indicators of compromise (analysis at scale).
• Configuration analysis: Hunting for misconfigured devices, including incorrect policies, overreaching access, endpoints without installed security controls, etc. (system hardening).
• Internal hunting: Identifying sensitive data, misconfigurations, plain text passwords, tokens stored or passed inappropriately, unauthorized access to critical systems, and abuse of user identities and service accounts (insider and environmental threats).
• External hunting: Focused on identifying instances of data, credentials leaks  and/or malware/hacking discovery using third party services such as VirusTotal, Shodan, Dark Web searches, and external surface scanning (external threat).

A crucial aspect of being successful in cybersecurity is the team’s adaptability to the ever-evolving threat landscapes. Security teams must consistently refine their methodologies and remain updated on emerging threats and evolving attack techniques.

Equally important is fostering collaboration among various security teams, including network security, incident response, SOC/SOAR, vulnerability management, and threat intelligence teams. By sharing insights and findings, these teams collectively enhance their ability to protect against cyber threats effectively.

On the Hunt | SolarWinds SERV-U Vulnerability

Thanks to our WatchTower threat hunting team we can see an example of valuable threat hunting based on traditional methodologies when we take a look at the timeline and story of how exploiting the SolarWinds SERV-U Vulnerability was proven to be connected to the download, decryption, and execution of Cobalt Strike. This proves that a structured methodology supported with data analysis at scale and focused threat hunting can be successful in identifying exploitation of known vulnerabilities with traditional attack methods.

1. In July 2021, SolarWinds released an advisory on Serv-U version 15.2.3. Microsoft stated that this CVE was used in limited, targeted attacks. After just a few days CISA also released an advisory that this vulnerability may allow a remote attacker to take control of an infected system even though it had not been identified performing such activities in the wild.
2. Using Deep Visibility queries and Vigilance MDR analyst investigation, the SentinelOne WatchTower threat hunting team spotted abuse of the Solarwinds Serv-U vulnerability in an educational institution exhibiting anomalous behaviors such as spawning unusual child processes.
   
3. The vulnerable Serv-U secure FTP launched the command prompt and powershell interface to connect to a remote C2 IP of http://179[.]60[.]150[.]32/login. The C2 IP address was live and served the next level of encoded commands as of August 26, to further decrypt and execute Cobalt Strike in memory.
   
4. The SentinelOne Agent successfully blocked and mitigated this attack before it could infect the target machine. Afterwards, it kicked off remediation and patching activities that likely should have been performed before the vulnerability was ever exploited.
5. The idea is that a modern futuristic paradigm to threat hunting should get ahead of these types of threats and show attempts before they are able to get even this far.

Modern Futuristic Paradigm

The vision of a modern and futuristic threat hunting paradigm involves leveraging advanced technologies and methodologies to enhance security operations and stay ahead of cyber threats. In this paradigm, threat hunting becomes a central focus of Security Operations Centers (SOCs) augmented by service providers and threat experts. Internal and external teams continually conduct research on known and emerging threats, vulnerabilities, and attack techniques for attribution and correlation. This research is then operationalized to proactively identify potential threats and vulnerabilities within an organization’s environment.

Automation plays a pivotal role in this vision. Routine and repetitive tasks are automated to free up security analysts’ time for more strategic activities. Automation can include the automatic collection and analysis of threat intelligence, the correlation of security events, extraction of indicators of compromise (IoCs), and the orchestration of incident response workflows.

Adapting to Changing Security Threats

Adaptive threat hunting is a dynamic approach to proactive anomaly discovery that evolves alongside the ever-changing threat landscape. It recognizes that threats can emerge from various sources, and it goes beyond traditional threat hunting by incorporating offensive inputs, novel research, and a range of hunting strategies.

Threat hunting should include new and real time strategies that address emerging threats in the present; Retroactive hunts, which delve into historical data for hidden threats; Artifact-based searches, which examine digital traces left by attackers; and performing Hunts of Hunts, which involves identification of the overarching strategies and tactics employed by adversaries based on chained detections, with a multi-directional approach for threat attribution.

By embracing these adaptable methods, organizations can strengthen their security postures and better protect against a diverse array of threats. Over this series of blogs we will introduce a modern approach and futuristic paradigm to threat hunting that allows us to stay ahead of the adversary and explore previous hunts analyzing the factors that made them successful. Key aspects of this vision include:

• Multi-directional Approach: Variety of techniques, data sources, and methodologies to comprehensively understand an organization’s security posture leveraging diverse telemetry, sweeps and scans, and LFO (Low Frequency of Occurrence) statistical analysis you can identify patterns in seemingly unrelated data.
• Chained Detections: Involves a sequence of automated tasks triggered by an initial detection to triage and enrich telemetry data progressively from disparate data sources. This approach is a proactive and sophisticated way to uncover and respond to complex threats and threat actors.
• AI and Machine Learning: Artificial Intelligence (AI) and Machine Learning (ML) methods and techniques are integrated into threat hunting processes. These technologies can analyze vast amounts of data, identify anomalies, and detect subtle patterns indicative of potential threats. Machine learning can identify potential threats in reams of seemingly innocuous data by autonomously formulating patterns that lead to malicious behavior, based on statistical event mapping.Generative AI threat hunting can also help security teams detect and respond to threats faster and more accurately.
• Threat Intelligence Integration: Threat hunting teams integrate internal and external intelligence feeds into their platforms. Threat intelligence platforms and Information Sharing and Analysis Centers (ISACs) facilitate the exchange of threat data and insights, enabling organizations to benefit from collective knowledge. This includes information on known threats, indicators of compromise (IoCs), and emerging attack tactics.
• Adaptive Continuous Hunting: The security posture is adaptive and responsive. Threat hunting is not a one-time event but an ongoing, adaptive process. Security controls, policies, and threat hunting strategies are continuously adjusted based on evolving threats and the organization’s risk profile.
• Incident Response Orchestration: Automated incident response orchestration and playbooks (such as those configurable in a SOAR or XDR platform) are used to streamline and accelerate response efforts. Security teams can respond to threats more effectively and consistently with predefined workflows.

A Modern Approach to Hunting

In this modern paradigm, the SOC’s role expands beyond reactive incident response to include proactive research, automated processes, and advanced technologies. There have been many attempts to tackle this problem, including Risk Based Alerting, clustering, baselining, allowlisting/denylisting, data normalization, tokenization, and additional data enrichment strategies, but there are more highly effective methods that we will continue to describe over the next few blog posts.

The goal is to create a resilient security posture capable of defending against a constantly evolving threat landscape with minimal impact on internal resources and security operations.

Akira Ransomware Campaign Highlights

Now we will show the results of a more extensive and modern hunt that yielded threat attribution and a higher level of fidelity and accuracy in identifying the risk and threat actors involved in an attack highlighting the Akira Ransomware Campaign. This was identified by our Vigilance DFIR team working closely with our WatchTower threat hunting team.

Akira ransomware operations were first observed in early 2023, with all the features and assets we expect from modern ransomware familles. This included a victim blog site, multi-platform payloads, and even a retro style branding. Once access is achieved, Akira focuses on stealing confidential documents, destroying backups, disabling security settings, and performing other nefarious activities leading to the extortion of the victim for a handsome ransom.

The following steps lay out the effectiveness of a multi-directional approach that is adaptive and leverages different sources of data intelligence to paint a full picture of a threat actor.

1. During an Akira incident we identified the group was downloading RustDesk, a remote access and remote control software available for different platforms. Notably, Akira had previously been associated with AnyDesk for persistence and C2 tasks, not the RustDesk RMM tool. It then proceeds to create services, disable the firewall, and allow remote connections from Akira operators.
   

2. Internal recon leveraged Advanced IP Scanner and NETSCAN.EXE for mapping the network along with winscp for data exfiltration. They proceeded to access and manipulate SQL databases for the purpose of mapping users, data, and environment.
   

3. The threat actor even went so far as to disable LSA (Local Security Authority) settings which help defend Windows users against credential theft by preventing untrusted code from being injected into the LSASS.exe process and disabling protections built into Windows.
   

4. While cross-referencing Akira victim blog data with Shodan data it looked like CISCO VPN gateways that belonged to targeted victim organizations were also listed. This suggests that Akira ransomware operators may be exploiting a vulnerability in Cisco VPN software to gain initial access. There is also evidence that stolen credentials acquired via IABs (Initial Access Brokers).
   

5. Detecting and blocking this ransomware at the endpoint is the last line of defense. Once a detection is made, that should then turn into actionable intelligence that leads to other investigative directions that help us anticipate attackers’ next steps.

In this case, new tactics and techniques were identified and attributed to a threat actor based on adaptive, continuous threat hunting and external threat analysis. Researching, analyzing, understanding, and hunting for this attack chain enabled our hunters to proactively hunt for similar activity and block it before it became a true threat. Subsequent hunts in other organizations allowed us to detect early and prevent many breaches.

Conclusion

The importance of creating business value through threat hunting in today’s complex and rapidly evolving cybersecurity landscape cannot be overstated. With the proliferation of AI, Cloud, SaaS, IoT, containers, growing market share of macOS, and omnipresent mobile devices, along with the challenges posed by regulated markets and remote work environments, organizations are facing a detection world that has become incredibly intricate. In response to these complexities, it makes strategic sense for many organizations to outsource advanced threat hunting and analysis to specialized security vendors to augment their own capabilities.

The reasons for this shift towards outsourcing this function to expert threat hunters as opposed to having a dedicated threat hunting team are compelling. Detection engineering has matured, and organizations are recognizing that a ‘build’ mentality often leads to playing catch-up with emerging threats. By ‘buying’ the expertise of a security vendor, organizations can leverage the vendor’s multidisciplinary team, which is exposed to new threats and tactics on a daily basis, often well ahead of in-house security teams. This proactive approach reduces the organization’s exposure to risks and accelerates threat response capabilities.

Additionally, the total cost of ownership is significantly lower with a managed service compared to maintaining a salaried internal team with limited expertise. Reliable threat hunting partners provide access to a larger pool of specialized skills as well as access to large data sets of rich telemetry across disparate endpoints and malware tactics. Internal staff can be augmented to enhance the organization’s security posture without the overhead of hiring and training more personnel. The risk of not conducting threat hunting is clear, and even with security tools that offer tremendous quantities of telemetry, it’s essential to have experts process this data to maximize its benefits, predicting the attack instead of just preventing it.

Learn More About WatchTower

For enterprises looking for a threat hunting partner to help them implement a robust methodology to stand up to emergent threats, SentinelOne’s WatchTower provides threat hunting experts equipped with the latest threat intelligence and  AI/machine learning algorithms.

Today, customers can use WatchTower to achieve real-time and retroactive detections of anomalous activity across their enterprise to proactively address evolving threats and strengthen their security posture. Learn more about what WatchTower can do for your enterprise here.

Special thanks to the entire WatchTower, Vigilance, and DFIR teams for contributions in findings, analysis, and content.