As a security analyst investigating security incidents or threat hunting, you often must navigate among multiple executed queries and their respective results (in other words, investigation steps). You also need to manage the evidence you gather during investigations and reuse it when building additional queries. All this must be as fast and seamless while maintaining the investigation context.

Navigating investigations and threat hunting can involve saving executed queries and found evidence in unstructured text documents that can quickly lose their context. Manually documenting investigation steps can lead to human errors, like forgotten or wrongly documented queries or evidence, which will result in missed threats and wasted time. As the investigation grows and the different queries accumulate, it becomes harder to maintain the throughlines among the different branches.

The Dynatrace Security Investigator app provides branching navigation, making it easy to track your path through investigations.

Navigate in any direction within your investigation

Investigations in Grail are conducted using the Dynatrace Query Language (DQL). For multiple investigation steps, the Security Investigator provides a representation of the steps in a tree-like view. With this visualization, you can keep track of your investigation and navigate to previous steps and respective results in the investigation history—all in context, with no need to re-execute anything.

Every new DQL query execution creates a new descending node while keeping the integrity and hierarchy of previous queries and results unchanged.

Easily track threat-hunting twists and turns

Threat hunting is a nonlinear process. When looking for the unknown, you often stumble upon an unexpected thread of evidence that sparks a new idea and takes the investigation in a new direction. To support this nonlinear approach, we’ve created the investigation branching feature. When modifying and re-executing a previous step, the app creates a new branch while keeping the original investigation path intact and visible in the query tree.

You can create any new branches for new ideas and hypotheses while retaining the ability to navigate back to your initial train of thought. With the ability to customize and name branches, you can simplify locating historical steps without having to click through them individually.

The final visual representation provides a comprehensive overview of the entire investigation. You can see in a structured way how you reached your conclusions without needing to manually document the investigation flow.

With Security Investigator, you can always find the last working query, which was easy to lose in the past.

Character precision on a petabyte scale

Security Investigator increases the speed of investigation flows and the precision of evidence, leading to higher efficiency and faster results. Both outcomes are game-changers in time-critical situations, where answers are expected “yesterday.”

View raw content details when you need them

Data ingested into Grail is kept in its original format. A schema is applied only during query execution, or as we call it, “schema-on-read.” This makes data immune to format changes and enables you to choose your data structure based on your use cases.

The Security Investigator app enables you to view all data in its original format, regardless of the content, including characters that other tools might hide or not interpret. Enabling the multi-line content details view shows the stack traces with their line breaks in their original form, enabling you to understand the data much faster.

Find the needle in the “needle stack”

Finding a small metallic object in a stack of non-magnetic objects is relatively easy. Finding a particular object among millions of similar objects is trickier. With Security Investigator’s flexible filtering, you can create accurate DQL query filters quickly and efficiently.

If you need to fetch raw content from Grail and filter the results based on a selection of the content, you don’t need to copy partial records and manually paste them into an appropriate filter. Instead, you can easily select the portion of the text string you need and add it to your DQL query directly from the context menu.

To add multiple values to a filter, you can either select individual values or a range of values and apply them as a single filter in your DQL query. Only the unique values from the selection are added to the query, so you don’t have to investigate unneeded values.

Use your findings to enrich your investigation

With Security Investigator evidence management, you can persist relevant findings by attaching them to your investigation. Whether it is a suspicious IP address or a potential indicator of compromise, you can attach it to the investigation and use it later.

You can create type-specific filters to enable fast and accurate results, either focusing only on suspicious activities or filtering out the safe events that don’t require your attention.

Continue hunting for the unknown

Security use cases such as threat hunting and incident analysis are nonlinear, evidence-driven activities in which you don’t know what you’re looking for. You’re not pursuing a specific target; rather, you’re looking for the unknown. With Security Investigator, Dynatrace enables you to be more efficient, investigate faster, and be more precise.

Use Dynatrace Security Investigator for faster and more precise evidence-based security investigations.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.

Today’s security analysts have no easy job. Not only are cyberattacks increasing, but they’re also becoming more sophisticated, with tools such as WormGPT putting generative AI technology in the hands of attackers. As a result, analysts are turning to AI and TTP-based threat-hunting techniques to uncover how attackers are trying to exploit their environments.

While AIOps with generative AI will certainly empower security teams to mitigate threats faster and with greater precision, attackers will just as certainly utilize the same technology to create novel malware, more convincing phishing campaigns, and uncover high-risk zero-day vulnerabilities quicker.

Not only that, teams struggle to correlate events and alerts from a wide range of security tools, need to put them into context, and infer their risk for the business. But the industry as a whole is still hampered by a ubiquitous tool sprawl to achieve that critical mission under a barrage of alert noise.

In this blog post, we’ll use Dynatrace Security Analytics to go threat hunting, bringing together logs, traces, metrics, and, crucially, threat alerts. We use the power of DQL on Grail to derive high-level attacker tactics, techniques, and procedures (TTPs), which are much easier to interpret and act upon.

TTP-based threat hunting: Tactics, techniques, procedures

At Dynatrace, we don’t want to bombard you with alert noise and uncorrelated warnings. Instead, we want to focus on detecting and stopping attacks before they happen: In your applications, in context, at the exact line of code that is vulnerable and in use. But even when an attack happens, Dynatrace detects and blocks them in real time while providing you with rich technical details on the concrete attack procedure. Procedures describe the specific technical details that an adversary used to carry out an attack, for example, what script they ran to exploit a weakness.

When investigating advanced cyberattacks, it’s helpful to map attack procedures to attack techniques. Techniques describe the tactical goal an adversary is pursuing by executing a specific procedure. One of the most critical attack techniques within the MITRE ATT&CK® knowledge base of adversary tactics and techniques — and one example of what Dynatrace can prevent, detect, and block in real-time — is attack technique T1190, “Exploit Public-Facing Application”.

Public-facing applications can be an initial access vector an attacker could exploit to gain entry into a system. Attack tactics describe why an attacker performs an action, for example, to get that first foothold into your network.

Thinking in terms of tactics, techniques, and procedures (TTPs) brings many benefits. For example, security analysts can more easily stitch together advanced cyberattacks on an abstract level. Likewise, operation specialists can prioritize their efforts on monitoring the highest-risk tactics, and executives can better communicate the business risk.

Threat hunting and analyzing threat alerts with Dynatrace Security Analytics and Grail

Dynatrace offers Runtime Application Protection to detect a wide range of injection attacks in your applications. However, our customers often want to augment the data Dynatrace provides with data from third-party tools. Customers also want to carry out their own analysis tailored to specific use cases and forensic needs.

Dynatrace Grail is a data lakehouse that provides context-rich analytics capabilities for observability, security, and business data. You may also ingest additional data into our unified intelligence platform: One popular choice to gather fine-grained security data is Falco. Falco is an open-source, cloud-native security tool that utilizes the Linux kernel technology eBPF, to generate fine-grained networking, security, and observability events.

In the following sections, we demo the following:

1. Introduce Unguard, our insecure cloud-native microservices demo application.
2. Install Falco in AWS EKS to gather security-relevant events from all the happenings in Unguard.
3. Ingest those Falco events into Dynatrace Grail using falcosidekick.
4. Query Falco events in Dynatrace Grail, map them to TTPs, and conduct structural multi-step attack detection.

In other words, we find attacks that are composed of multiple steps by using TTPs and Dyntrace Smartscape for DQL in a way that eliminates alert noise.

First, Dynatrace OneAgent will automatically monitor and trace our infrastructure and communicate with Dynatrace. Second, we will enrich our data in the Grail data lakehouse by also ingesting Falco events using falcosidekick.

Setting up our TTP-based threat-hunting demo environment

Before we start threat hunting, we’ll first walk through how to set up the demo environment.

Introducing Unguard, our insecure cloud-native demo app

As our playground, we introduce Unguard, a microblogging demo application that embodies the challenges of modern cloud-native environments. It consists of eight services, written in at least four different languages, with countless vulnerabilities and misconfigurations. To keep it real, we have a load generator that creates benign traffic. It also generates OpenTelemetry traces.

Unguard was first introduced at DEFCON 31 by our colleagues Simon Ammer and Christoph Wedenig.

For the demonstration in this blog post, we want to deploy Unguard in AWS EKS and hunt for attacks within that environment. You can easily play around with Unguard by installing its Helm chart:

helm install unguard \ 
  oci://ghcr.io/dynatrace-oss/unguard/chart/unguard \ 
  --wait --namespace unguard --create-namespace

(Please read the Unguard README for detailed and up-to-date instructions)

This demo assumes your Kubernetes cluster is already monitored by Dynatrace. For instructions, see Set up Dynatrace on Kubernetes.

Deploy Falco and falcosidekick in AWS EKS

You can install Falco in various ways. For this demo, we installed it with the Helm chart in our AWS EKS cluster:

helm repo add falcosecurity https://falcosecurity.github.io/charts 
helm repo update 
helm install falco falcosecurity/falco --namespace falco --create-namespace

(See the Falco README for detailed and up-to-date instructions)

Next, we set up falcosidekick, which is a daemon that forwards Falco events to many possible outputs. We’re proud to announce that, with Falco version 2.29, currently in pre-release, you can now also use Dynatrace as an output.

You can use this minimal values.yaml configuration file for the Helm chart:

# values.yaml 
 
falcosidekick: 
  enabled: true 
  image: 
    tag: 2.29.0-rc.1 
  config: 
    # as of 2023-08-02, this feature is still a pre-release so we 
    # have to manually override the environment variables for now 
    extraEnv: 
      - name: DYNATRACE_APITOKEN 
        value: dt0c01.EXAMPLE_TOKEN_REPLACE_THE_ENTIRE_STRING 
      - name: DYNATRACE_APIURL 
        value: https://ENVIRONMENTID.live.dynatrace.com/api

(Please read the Helm chart README for detailed and up-to-date instructions)

We insert the apitoken we generated within Dynatrace and grant the token the scope logs.ingest. See the topic Dynatrace API – Tokens and authentication to learn more about creating tokens. As the apiurl, use the following:

Dynatrace SaaS:

https://ENVIRONMENTID.live.dynatrace.com/api

Dynatrace Managed:

https://YOURDOMAIN/e/ENVIRONMENTID/api

See the topic Environment ID to learn more about environment IDs.

Finally, we update the Falco Helm chart with this new configuration:

helm upgrade falco falcosecurity/falco -f values.yaml

If everything worked out well (check the pod logs otherwise), we are now able to successfully query Falco events with DQL. To verify, we open a new Notebook and see how Dynatrace automatically infers the fields from our events already:

fetch logs, from:now() - 5m 
| filter (event.provider == "Falco")

Observing TTPs using Dynatrace Security Analytics

For the sake of this demonstration, our internal red team unleashed a novel attack on our Unguard application. The attack lit up our Falco deployment with more than 100,000 events in 24 hours, more than 3,000 of them critical.

As security analysts, we know we can’t find sophisticated attacks by manually scrolling through thousands of audit logs and events. We need automation, full contextual knowledge of our infrastructure, and very often, domain-specific expertise from security analysts.

To get an initial overview, we can use DQL on Grail to visualize what MITRE techniques Falco observed in our infrastructure over the past 72 hours. We can summarize events using mitre.tactic or mitre.technique. These fields exist on many Falco alerts and are automatically ingested by the Dynatrace output of falcosidekick. We can explore the distribution of techniques with this query:

fetch logs, from:now() - 72h 
| filter event.provider == "Falco" and isNotNull(mitre.technique) 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize event_count = count(), by:{mitre.technique}

In this example, we also observe that we can attribute most events to the following MITRE techniques:

• T1548.001
• T1083
• T1565
• T1055.008

After manually investigating these alerts, however, we conclude they’re noisy false positives. Some of our applications were treating environment variables in an insecure way or communicating with the Kubernetes API server with improperly configured service accounts. Therefore, we filtered them out with DQL.

Observability and context: Attributing reconnaissance activity to TTPs using distributed traces

So far in our TTP-based threat hunting, we’ve utilized Dyntrace Security Analytics to visualize ingested alerts from third-party tools.

But truly magical things arise when we combine this with the rich and high-quality observability data that our customers have valued since the beginning of Dynatrace. Using observability data, we can close an important security-relevant gap. Attackers often probe systems using automated scanning tools. Their many access attempts leave behind a lot of traces. Dynatrace PurePath is one of the core platform technologies that captures and analyzes those distributed traces across an entire infrastructure.

Attack sub-technique T1595.003 “Active Scanning: Wordlist Scanning” describes how attackers use scanners to learn about the many endpoints an application might expose to the internet. Typically, they use large lists with well-known path names, where many of them could be potentially vulnerable. Such wordlists often contain common path names, such as wp-admin, .git, or .htaccess. The following query looks for five indicative files and expresses how many of them match with the new recon.confidence field we set up to track wordlist-based scanners. The more matches, the more confident we can be that these requests came from a wordlist-based scanner:

fetch spans, from:now() - 72h 
| filter in(http.target, {"/wp-admin", "/.git", "/.htaccess", "/.ssh", "/cgi"}) 
| summarize { 
    recon.confidence = countDistinct(http.target) / 5, 
    recon.first_seen = min(timestamp), 
    recon.last_seen = max(timestamp) 
  }, by:{host.name, k8s.container.name, k8s.namespace.name, k8s.pod.name} 
| fieldsAdd mitre.technique = "T1595.003", mitre.tactic = "mitre_reconnaissance" 
| filter recon.confidence > 0.5

Indeed, it did find some reconnaissance attempts! This query scanned 2.5 million spans in less than 50 ms and reduced them to three comprehensible TTP records. With a conventional database, a query that scans millions of records would take many seconds to complete and require that we structure our queries up front. But Grail completes the search across millions of records in milliseconds, and automatically parses and infers the structure for us so we can just start writing queries directly.

We now know that the Envoy proxy in our environment most likely got scanned by an attacker. Let us bring all the bits and pieces together in the next sequence.

Structural multi-step attack detection with Dynatrace Security Analytics

Attackers typically perform many small steps to achieve their mission. Security experts like to think in terms of so-called kill chains, which describe the many stages of an attack. When you work with TTPs, the attack tactics represent those stages. While the MITRE ATT&CK® knowledge base describes as many as 14 tactics, we can distill this into three broad categories:

1. Land – First, hackers investigate their target, looking for an initial way to gain access and establish a foothold in your system.
2. Expand – Then, hackers typically try to escalate their privileges and move laterally within your system, compromising neighboring hosts.
3. Execute – Finally, they find their target and execute their mission.

Next in our demonstration of TTP-based threat hunting with Dynatrace Security Analytics, we’re going to show you a simple but effective strategy that can uncover such advanced attacks: Structural multi-step attack detection. This strategy is structural since it utilizes Smartscape for DQL to take the topological relationship of events into account when hunting for attacks that are composed of multiple steps.

The following DQL query looks for the filtered Falco alerts, and for each Kubernetes pod, it records how many distinct tactics and techniques we just observed. This way, we’re not just looking at whatever pod was the noisiest, but instead, which pod generated alerts from the most tactics and techniques. The more tactics and techniques, the higher the chances that an attacker carried out a full kill chain on that pod.

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize { 
    num_tactics = countDistinct(mitre.tactic), 
    num_techniques = countDistinct(mitre.technique), 
    tactics = collectDistinct(mitre.tactic), 
    techniques = collectDistinct(mitre.technique) 
  }, by:{k8s.pod.name} 
| filter num_tactics > 1 and num_techniques > 1 
| sort num_tactics desc, num_techniques desc

This result is highly interesting and confirms our previous suspicion. There is one instance of the Envoy proxy that captured alerts for the following techniques:

• TA0002 “Execution”
• TA0003 “Persistence”
• TA0005 “Defensive Evasion”
• TA0006 “Credential Access”
• TA0011 “Command and Control”
• TA0010 “Exfiltration”

Further, the Dynatrace spans we looked at in the previous sequence that explored wordlist scanning indicated TA0043 “Reconnaissance”. This query just scanned through more than 23 million records in 300 ms, providing us with an abstract description of a full kill chain.

But we don’t stop here. We can drill down and observe the individual steps our attacker has taken:

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| filter k8s.pod.name == "unguard-envoy-proxy-666464f76d-5p26f" 
| fields timestamp, event.name, mitre.technique, content.output_fields.proc.cmdline 
| sort timestamp asc

In the result, we see the records that explain the attack procedure in detail. The above screenshot shows only an excerpt of all 47 records. Here’s what we learned about our attacker’s steps:

• Scanned our Envoy proxy with a well-known wordlist, as we learned by mining the traces for wordlist entries.
• Launched a Perl-based reverse shell on Envoy, indicated by the perl command opening a socket, giving them full code execution access.
• Downloaded a couple of binaries like nmap and nc, indicated by the curl command that pulled them from the internet.
• Scanned our internal network with nmap.
• Exfiltrated large volumes of data from our Redis database, indicated by the queries to Redis that request all keys with the KEYS * command
• Tried to cover their tracks by deleting the shell history, indicated by the rm /home/envoy/.bash_history command

Isn’t this a truly elegant way to hunt for attacks?

TTP-based threat hunting with context-rich observability and security analytics

This demonstration shows how modern attack detection strategies become a reality with context-rich security analytics on a unified observability and security platform. With this approach, you can do the following:

• Utilize observability data to capture security-relevant reconnaissance alerts and map them to TTPs.
• Enrich the Dynatrace platform with more data of your own, such as ingesting Falco alerts into Grail.
• Use DQL and Grail to find the needle in the haystack to scan tens of millions of records in milliseconds to identify the chain of only a handful of events that exposed an attacker and the exact methods they used.

For another great demonstration, we recommend reading the blog post Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail by Liisa Tallinn.

The post TTP-based threat hunting with Dynatrace Security Analytics and Falco Alerts solves alert noise appeared first on Dynatrace news.

With up to 70% of security events going uninvestigated, security analysts need all the help they can get. After a security event, many organizations often don’t know for months (or even years) when why or how it happened. This represents a significant risk, with the same attack vector repeatedly getting exploited if a vulnerability is not resolved on time. The massive volumes of log data over months, sometimes years, of a breach have made this a complicated and expensive problem to solve.

A traditional log-based SIEM approach to security analytics may have served organizations well in simpler on-premises environments. But this limited approach causes challenges in today’s hybrid multicloud reality. With the rising complexity of cloud-native environments, manual investigation and response are too slow and inaccurate. Teams must evolve to continuously create reliable, automated responses based on precise data-driven insights. Comprehensive datasets, including topology and runtime context, can make it easier to find the needle in the haystack and understand the significance of events and vulnerabilities.

Experience with the recent MOVEit vulnerability illustrated some of the key incomplete data challenges organizations face when trying to find definitive answers to questions like “were we exploited?” and “was any sensitive data stolen?” Relying only on logs to find indicators of compromise (IoC) is no longer effective, especially for application attacks, because logs simply don’t contain all the clues. As our experience with MOVEit shows, IoCs that remained hidden in logs alone quickly revealed themselves with observability runtime context data, such as metrics, traces, and spans.

Extending application security protection to Security Analytics

With Dynatrace Runtime Vulnerability Analytics, Dynatrace customers have reduced the amount of time and effort spent on identifying and prioritizing vulnerabilities in both custom code and third-party code. Additionally, Runtime Application Protection provides the ability to protect from attacks while giving development teams much-needed time to remediate these vulnerabilities.

Dynatrace Security Analytics now extends these capabilities by combining predictive and causal AI techniques to help security analysts and architects investigate suspected or detected attacks and create automated response workflows. Security Analytics combines Dynatrace platform capabilities (such as Grail data lakehouse and AutomationEngine) with analytics capabilities (such as Dynatrace Pattern Language (DPL) architect) that make life easier for security analysts.

In an industry first, customers can conduct threat detection, forensics, and incident response use cases based on a combined security and observability dataset enhanced by topology context. Grail can deal with any data, be it OpenTelemetry data or large-scale amounts of security data. Dynatrace OneAgent automatically discovers relevant observability and topology data across complex environments, which provides context and rich data. This is a differentiated and more evolved approach than simply using logs, maximizing the precision, breadth, and depth of insights.

Security Analytics and automation deal with unknown-unknowns

With Security Analytics, analysts can explore the unknown-unknowns, facilitating queries manually in an ad hoc way, or continuously using automation. This approach addresses classic security-driven log analysis and SIEM use cases, and includes threat hunting and looking for anomalies or IoCs.

• Observability (runtime) context: Utilizing contextualized observability data, you can combine traces, logs, and metrics with security events using AI-driven analysis. This combination elevates use cases that were historically conducted predominantly on only log data. As a result, not only can you understand, for example, that someone accessed a database, but also from where they came, exactly what they accessed, and to where they exported the data–to the level that we know the exact database query statement.
• Automation: Automation plays a crucial role in dealing with the complexities and scale of cybersecurity. By automating routine tasks, such as data collection, analysis, and incident response, organizations can improve their ability to detect and respond to unknown unknowns in a timely manner. With automated processes, you can rapidly identify patterns, correlations, and deviations, allowing security teams to focus on investigating and mitigating emerging threats. Additionally, automation enables faster threat containment, reduces response times, and minimizes the potential for human error.
• Advanced analytics: Advanced analytics techniques, including causal Davis AI and generative AI, facilitate human interaction, enable organizations to analyze vast amounts of data, and extract valuable insights. By applying advanced analytics to security and contextualized observability data, organizations can uncover hidden patterns, trends, and anomalies that may indicate the presence of unknown unknowns that may go undetected by traditional security measures. Advanced analytics empowers organizations to detect and respond to emerging threats proactively, staying ahead of cyber adversaries.

What can you do with Dynatrace Security Analytics?

Here are some samples of what you can do today with Dynatrace Security Analytics.

• Threat hunting: Dynatrace Security Analytics provides analysts with unique capabilities that enhance productivity by collaboratively investigating suspected attacks, automating response, and implementing proactive threat-hunting strategies. Notebooks enable teams to create playbooks to iteratively construct complex queries, review results, and refine to quickly zoom on IoCs. DPL (Dynatrace Pattern Language) simplifies extracting information out of varied log formats without needing to write complicated regex. Analysts can use AutomationEngine to continuously monitor and respond to IoCs.
• Incident response: With cost-effective, long-term data retention allowing teams to go back in time for months or years to identify the root cause of the attack. The combination of data with retained context and lightning-fast queries empowers analysts to identify IoCs, reconstruct events, and determine next steps in record time. Analysts can leverage AutomationEngine to continuously monitor and respond to future attacks.
• Log storage and data retention: As regulations grow more stringent, data retention requirements and costs can quickly mount. Dynatrace Grail data lakehouse offers a scalable, affordable way to store data long term while keeping all data always available for dashboarding and analysis.

What’s next for Security Analytics?

Security Analytics with Davis® AI, the Grail data lakehouse, AutomationEngine, and Notebooks are all available for customers to use today.

In the coming months, we look forward to further enhancing analyst productivity with Davis CoPilot generative AI and security-specific user experiences. Davis CoPilot will enable natural language queries, suggest CISO dashboards to track progress, and auto-create security incident response workflows using AutomationEngine.

Leave the beaten path of traditional security tooling for a future with unified observability and security

Proactive incident response is based on understanding what’s happening at runtime in real-time across the full stack by identifying suspicious activities that may lead to potential breaches. This modern approach puts security analysts in the driver’s seat. A coordinated organizational approach to patching vulnerabilities or initiating incident response before an actual breach occurs increases speed, and reduces costs, and accelerates innovation. The overall reduced risk of falling victim to cyber-crimes readies organizations utilizing Dynatrace’s unified observability and security platform for the projected increase in cyber-attacks.

The post Dynatrace unveils Security Analytics to elevate threat detection, forensics, and incident response appeared first on Dynatrace news.